Callbacks that take decisions depending on the evolution of metrics during training

Tracking Callbacks

This module regroups the callbacks that track one of the metrics computed at the end of each epoch to take some decision about training. To show examples of use, we'll use our sample of MNIST and a simple cnn model.

path = untar_data(URLs.MNIST_SAMPLE)
data = ImageDataBunch.from_folder(path)

class TerminateOnNaNCallback[source][test]

TerminateOnNaNCallback() :: Callback

No tests found for TerminateOnNaNCallback. To contribute a test please refer to this guide and this discussion.

A Callback that terminates training if loss is NaN.

Sometimes, training diverges and the loss goes to nan. In that case, there's no point continuing, so this callback stops the training.

model = simple_cnn((3,16,16,2))
learn = Learner(data, model, metrics=[accuracy])
learn.fit_one_cycle(1,1e4)
epoch train_loss valid_loss accuracy time
1 nan nan 0.495584 00:02

Using it prevents that situation to happen.

model = simple_cnn((3,16,16,2))
learn = Learner(data, model, metrics=[accuracy], callbacks=[TerminateOnNaNCallback()])
learn.fit(2,1e4)
0.00% [0/2 00:00<00:00]
epoch train_loss valid_loss accuracy time

Interrupted
Epoch/Batch (0/5): Invalid loss, terminating training.
Epoch/Batch (0/6): Invalid loss, terminating training.

Callback methods

You don't call these yourself - they're called by fastai's Callback system automatically to enable the class's functionality.

on_batch_end[source][test]

on_batch_end(last_loss, epoch, num_batch, **kwargs:Any)

No tests found for on_batch_end. To contribute a test please refer to this guide and this discussion.

Test if last_loss is NaN and interrupts training.

on_epoch_end[source][test]

on_epoch_end(**kwargs:Any)

No tests found for on_epoch_end. To contribute a test please refer to this guide and this discussion.

Called at the end of an epoch.

class EarlyStoppingCallback[source][test]

EarlyStoppingCallback(learn:Learner, monitor:str='val_loss', mode:str='auto', min_delta:int=0, patience:int=0) :: TrackerCallback

No tests found for EarlyStoppingCallback. To contribute a test please refer to this guide and this discussion.

A TrackerCallback that terminates training when monitored quantity stops improving.

This callback tracks the quantity in monitor during the training of learn. mode can be forced to 'min' or 'max' but will automatically try to determine if the quantity should be the lowest possible (validation loss) or the highest possible (accuracy). Will stop training after patience epochs if the quantity hasn't improved by min_delta.

model = simple_cnn((3,16,16,2))
learn = Learner(data, model, metrics=[accuracy], 
                callback_fns=[partial(EarlyStoppingCallback, monitor='accuracy', min_delta=0.01, patience=3)])
learn.fit(50,1e-42)
8.00% [4/50 00:09<01:51]
epoch train_loss valid_loss accuracy time
1 0.677729 0.677514 0.560353 00:02
2 0.679820 0.677514 0.560353 00:02
3 0.679434 0.677514 0.560353 00:02
4 0.679893 0.677514 0.560353 00:02

100.00% [32/32 00:00<00:00]
Epoch 5: early stopping
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-9-4384b4e1abc4> in <module>
      2 learn = Learner(data, model, metrics=[accuracy], 
      3                 callback_fns=[partial(EarlyStoppingCallback, monitor='accuracy', min_delta=0.01, patience=3)])
----> 4 learn.fit(50,1e-42)

~/fastai/fastai/basic_train.py in fit(self, epochs, lr, wd, callbacks)
    180         if defaults.extra_callbacks is not None: callbacks += defaults.extra_callbacks
    181         fit(epochs, self.model, self.loss_func, opt=self.opt, data=self.data, metrics=self.metrics,
--> 182             callbacks=self.callbacks+callbacks)
    183 
    184     def create_opt(self, lr:Floats, wd:Floats=0.)->None:

~/fastai/fastai/basic_train.py in fit(epochs, model, loss_func, opt, data, callbacks, metrics)
     97                                        cb_handler=cb_handler, pbar=pbar)
     98             else: val_loss=None
---> 99             if cb_handler.on_epoch_end(val_loss): break
    100     except Exception as e:
    101         exception = e

~/fastai/fastai/callback.py in on_epoch_end(self, val_loss)
    287         self.state_dict['last_metrics'] = [val_loss] if val_loss is not None else None
    288         self.state_dict['epoch'] += 1
--> 289         self('epoch_end', call_mets = val_loss is not None)
    290         return self.state_dict['stop_training']
    291 

~/fastai/fastai/callback.py in __call__(self, cb_name, call_mets, **kwargs)
    222         if call_mets:
    223             for met in self.metrics: self._call_and_update(met, cb_name, **kwargs)
--> 224         for cb in self.callbacks: self._call_and_update(cb, cb_name, **kwargs)
    225 
    226     def set_dl(self, dl:DataLoader):

~/fastai/fastai/callback.py in _call_and_update(self, cb, cb_name, **kwargs)
    213         "Call `cb_name` on `cb` and update the inner state."
    214         new = ifnone(getattr(cb, f'on_{cb_name}')(**self.state_dict, **kwargs), dict())
--> 215         for k,v in new.items():
    216             if k not in self.state_dict:
    217                 raise Exception(f"{k} isn't a valid key in the state of the callbacks.")

AttributeError: 'bool' object has no attribute 'items'

Callback methods

You don't call these yourself - they're called by fastai's Callback system automatically to enable the class's functionality.

on_train_begin[source][test]

on_train_begin(**kwargs:Any)

No tests found for on_train_begin. To contribute a test please refer to this guide and this discussion.

Initialize inner arguments.

on_epoch_end[source][test]

on_epoch_end(epoch, **kwargs:Any)

No tests found for on_epoch_end. To contribute a test please refer to this guide and this discussion.

Compare the value monitored to its best score and maybe stop training.

class SaveModelCallback[source][test]

SaveModelCallback(learn:Learner, monitor:str='val_loss', mode:str='auto', every:str='improvement', name:str='bestmodel') :: TrackerCallback

No tests found for SaveModelCallback. To contribute a test please refer to this guide and this discussion.

A TrackerCallback that saves the model when monitored quantity is best.

This callback tracks the quantity in monitor during the training of learn. mode can be forced to 'min' or 'max' but will automatically try to determine if the quantity should be the lowest possible (validation loss) or the highest possible (accuracy). Will save the model in name whenever determined by every ('improvement' or 'epoch'). Loads the best model at the end of training is every='improvement'.

model = simple_cnn((3,16,16,2))
learn = Learner(data, model, metrics=[accuracy])
learn.fit_one_cycle(5,1e-4, callbacks=[SaveModelCallback(learn, every='epoch', monitor='accuracy', name='model')])

Choosing every='epoch' saves an individual model at the end of each epoch.

!ls ~/.fastai/data/mnist_sample/models
learn.fit_one_cycle(5,1e-4, callbacks=[SaveModelCallback(learn, every='improvement', monitor='accuracy', name='best')])

Choosing every='improvement' saves the single best model out of all epochs during training.

!ls ~/.fastai/data/mnist_sample/models

Callback methods

You don't call these yourself - they're called by fastai's Callback system automatically to enable the class's functionality.

on_epoch_end[source][test]

on_epoch_end(epoch:int, **kwargs:Any)

No tests found for on_epoch_end. To contribute a test please refer to this guide and this discussion.

Compare the value monitored to its best score and maybe save the model.

on_train_end[source][test]

on_train_end(**kwargs)

No tests found for on_train_end. To contribute a test please refer to this guide and this discussion.

Load the best model.

class ReduceLROnPlateauCallback[source][test]

ReduceLROnPlateauCallback(learn:Learner, monitor:str='val_loss', mode:str='auto', patience:int=0, factor:float=0.2, min_delta:int=0) :: TrackerCallback

No tests found for ReduceLROnPlateauCallback. To contribute a test please refer to this guide and this discussion.

A TrackerCallback that reduces learning rate when a metric has stopped improving.

This callback tracks the quantity in monitor during the training of learn. mode can be forced to 'min' or 'max' but will automatically try to determine if the quantity should be the lowest possible (validation loss) or the highest possible (accuracy). Will reduce the learning rate by factor after patience epochs if the quantity hasn't improved by min_delta.

Callback methods

You don't call these yourself - they're called by fastai's Callback system automatically to enable the class's functionality.

on_train_begin[source][test]

on_train_begin(**kwargs:Any)

No tests found for on_train_begin. To contribute a test please refer to this guide and this discussion.

Initialize inner arguments.

on_epoch_end[source][test]

on_epoch_end(epoch, **kwargs:Any)

No tests found for on_epoch_end. To contribute a test please refer to this guide and this discussion.

Compare the value monitored to its best and maybe reduce lr.

class TrackerCallback[source][test]

TrackerCallback(learn:Learner, monitor:str='val_loss', mode:str='auto') :: LearnerCallback

No tests found for TrackerCallback. To contribute a test please refer to this guide and this discussion.

A LearnerCallback that keeps track of the best value in monitor.

get_monitor_value[source][test]

get_monitor_value()

No tests found for get_monitor_value. To contribute a test please refer to this guide and this discussion.

Pick the monitored value.

Callback methods

You don't call these yourself - they're called by fastai's Callback system automatically to enable the class's functionality.

on_train_begin[source][test]

on_train_begin(**kwargs:Any)

No tests found for on_train_begin. To contribute a test please refer to this guide and this discussion.

Initializes the best value.