Basic training functionality¶
basic_train
wraps together the data (in a DataBunch
object) with a PyTorch model to define a Learner
object. Here the basic training loop is defined for the fit
method. The Learner
object is the entry point of most of the Callback
objects that will customize this training loop in different ways. Some of the most commonly used customizations are available through the train
module, notably:
Learner.lr_find
will launch an LR range test that will help you select a good learning rate.Learner.fit_one_cycle
will launch a training using the 1cycle policy to help you train your model faster.Learner.to_fp16
will convert your model to half precision and help you launch a training in mixed precision.
The main purpose of Learner
is to train model
using Learner.fit
. After every epoch, all metrics will be printed and also made available to callbacks.
The default weight decay will be wd
, which will be handled using the method from Fixing Weight Decay Regularization in Adam if true_wd
is set (otherwise it's L2 regularization). If bn_wd
is False
, then weight decay will be removed from batchnorm layers, as recommended in Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. If train_bn
, batchnorm layer learnable params are trained even for frozen layer groups.
To use discriminative layer training, pass a list of nn.Module
as layer_groups
; each nn.Module
will be used to customize the optimization of the corresponding layer group.
If path
is provided, all the model files created will be saved in path
/model_dir
; if not, then they will be saved in data.path
/model_dir
.
You can pass a list of callback
s that you have already created, or (more commonly) simply pass a list of callback functions to callback_fns
and each function will be called (passing self
) on object initialization, with the results stored as callback objects. For a walk-through, see the training overview page. You may also want to use an application specific model. For example, if you are dealing with a vision dataset, here the MNIST, you might want to use the create_cnn
method:
path = untar_data(URLs.MNIST_SAMPLE)
data = ImageDataBunch.from_folder(path)
learn = create_cnn(data, models.resnet18, metrics=accuracy)
Model fitting methods¶
Runs the learning rate finder defined in LRFinder
, as discussed in Cyclical Learning Rates for Training Neural Networks.
learn.lr_find()
learn.recorder.plot()
Uses discriminative layer training if multiple learning rates or weight decay values are passed. To control training behaviour, use the callback
system or one or more of the pre-defined callbacks
.
learn.fit(1)
Use cycle length cyc_len
, a per cycle maximal learning rate max_lr
, momentum moms
, division factor div_factor
, weight decay wd
, and optional callbacks callbacks
. Uses the OneCycleScheduler
callback. Please refer to What is 1-cycle for a conceptual background of 1-cycle training policy and more technical details on what do the method's arguments do.
learn.fit_one_cycle(1)
See results¶
predict
can be used to get a single prediction from the trained learner on one specific piece of data you are interested in.
learn.data.train_ds[0]
Each element of the dataset is a tuple, where the first element is the data itself, while the second element is the target label. So to get the data, we need to index one more time.
data = learn.data.train_ds[0][0]
data
pred = learn.predict(data)
pred
The first two elements of the tuple are, respectively, the predicted class and label. Label here is essentially an internal representation of each class, since class name is a string and cannot be used in computation. To check what each label corresponds to, run:
learn.data.classes
So category 0 is 3 while category 1 is 7.
probs = pred[2]
The last element in the tuple is the predicted probabilities. For a categorization dataset, the number of probabilities returned is the same as the number of classes; probs[i]
is the probability that the item
belongs to learn.data.classes[i]
.
learn.data.valid_ds[0][0]
You could always check yourself if the probabilities given make sense.
It will run inference using the learner on all the data in the ds_type
dataset and return the predictions; if n_batch
is not specified, it will run the predictions on the default batch size. If with_loss
, it will also return the loss on each prediction.
Here is how you check the default batch size.
learn.data.batch_size
preds = learn.get_preds()
preds
The first element of the tuple is a tensor that contains all the predictions.
preds[0]
While the second element of the tuple is a tensor that contains all the target labels.
preds[1]
preds[1][0]
len(learn.data.valid_ds)
len(preds[0]), len(preds[1])
To get predictions on the entire training dataset, simply set the ds_type
argument accordingly.
learn.get_preds(ds_type=DatasetType.Train)
To also get prediction loss along with the predictions and the targets, set with_loss=True
in the arguments.
learn.get_preds(with_loss=True)
Note that the third tensor in the output tuple contains the losses.
Return the calculated loss and the metrics of the current model on the given data loader dl
. The default data loader dl
is the validation dataloader.
You can check the default metrics of the learner using:
str(learn.metrics)
learn.validate()
learn.validate(learn.data.valid_dl)
learn.validate(learn.data.train_dl)
Note that the text number on the top is the ground truth, or the target label, the one in the middle is the prediction, while the image number on the bottom is the image data itself.
learn.show_results()
learn.show_results(ds_type=DatasetType.Train)
Note that the number of predictions given equals to the batch size.
learn.data.batch_size
preds = learn.pred_batch()
len(preds)
Since the total number of predictions is too large, we will only look at a part of them.
preds[:10]
item = learn.data.train_ds[0][0]
item
batch = learn.data.one_item(item)
batch
learn.pred_batch(batch=batch)
For more details, refer to ClassificationInterpretation
Model summary¶
Test time augmentation¶
Applies Test Time Augmentation to learn
on the dataset ds_type
. We take the average of our regular predictions (with a weight beta
) with the average of predictions obtained through augmented versions of the training set (with a weight 1-beta
). The transforms decided for the training set are applied with a few changes scale
controls the scale for zoom (which isn't random), the cropping isn't random but we make sure to get the four corners of the image. Flipping isn't random but applied once on each of those corner images (so that makes 8 augmented versions total).
Gradient clipping¶
Mixed precision training¶
Uses the MixedPrecision
callback to train in mixed precision (i.e. forward and backward passes using fp16, with weight updates using fp32), using all NVIDIA recommendations for ensuring speed and accuracy.
Distributed training¶
Discriminative layer training¶
When fitting a model you can pass a list of learning rates (and/or weight decay amounts), which will apply a different rate to each layer group (i.e. the parameters of each module in self.layer_groups
). See the Universal Language Model Fine-tuning for Text Classification paper for details and experimental results in NLP (we also frequently use them successfully in computer vision, but have not published a paper on this topic yet). When working with a Learner
on which you've called split
, you can set hyperparameters in four ways:
param = [val1, val2 ..., valn]
(n = number of layer groups)param = val
param = slice(start,end)
param = slice(end)
If we chose to set it in way 1, we must specify a number of values exactly equal to the number of layer groups. If we chose to set it in way 2, the chosen value will be repeated for all layer groups. See Learner.lr_range
for an explanation of the slice
syntax).
Here's an example of how to use discriminative learning rates (note that you don't actually need to manually call Learner.split
in this case, since fastai uses this exact function as the default split for resnet18
; this is just to show how to customize it):
# creates 3 layer groups
learn.split(lambda m: (m[0][6], m[1]))
# only randomly initialized head now trainable
learn.freeze()
learn.fit_one_cycle(1)
# all layers now trainable
learn.unfreeze()
# optionally, separate LR and WD for each group
learn.fit_one_cycle(1, max_lr=(1e-4, 1e-3, 1e-2), wd=(1e-4,1e-4,1e-1))
Rather than manually setting an LR for every group, it's often easier to use Learner.lr_range
. This is a convenience method that returns one learning rate for each layer group. If you pass slice(start,end)
then the first group's learning rate is start
, the last is end
, and the remaining are evenly geometrically spaced.
If you pass just slice(end)
then the last group's learning rate is end
, and all the other groups are end/10
. For instance (for our learner that has 3 layer groups):
learn.lr_range(slice(1e-5,1e-3)), learn.lr_range(slice(1e-3))
Sets every layer group to trainable (i.e. requires_grad=True
).
Sets every layer group except the last to untrainable (i.e. requires_grad=False
).
A convenience method that sets layer_groups
based on the result of split_model
. If split_on
is a function, it calls that function and passes the result to split_model
(see above for example).
Saving and loading models¶
Simply call Learner.save
and Learner.load
to save and load models. Only the parameters are saved, not the actual architecture (so you'll need to create your model in the same way before loading weights back in). Models are saved to the path
/model_dir
directory.
show_doc(Learner.save)
learn.save("trained_model")
learn.save("trained_model", return_path=True)
learn = learn.load("trained_model")
Deploying your model¶
When you are ready to put your model in production, export the minimal state of your Learner
with
learn.export()
learn.export('trained_model.pkl')
path = learn.path
path
learn = load_learner(path)
learn = load_learner(path, fname='trained_model.pkl')
WARNING: If you used any customized classes when creating your learner, you must first define these classes first before executing load_learner
.
You can find more information and multiple examples in this tutorial
Other methods¶
Initializes all weights (except batchnorm) using function init
, which will often be from PyTorch's nn.init
module.
Uses MixUpCallback
.
You generally won't need to call this yourself - it's used to create the optim
optimizer before fitting the model.
learn.dl()
learn.dl(DatasetType.Train)
A Learner
creates a Recorder
object automatically - you do not need to explicitly pass it to callback_fns
- because other callbacks rely on it being available. It stores the smoothed loss, hyperparameter values, and metrics for each batch, and provides plotting methods for each. Note that Learner
automatically sets an attribute with the snake-cased name of each callback, so you can access this through Learner.recorder
, as shown below.
Plotting methods¶
This is mainly used with the learning rate finder, since it shows a scatterplot of loss vs learning rate.
path = untar_data(URLs.MNIST_SAMPLE)
data = ImageDataBunch.from_folder(path)
learn = create_cnn(data, models.resnet18, metrics=accuracy)
learn.lr_find()
learn.recorder.plot()
Note that validation losses are only calculated once per epoch, whereas training losses are calculated after every batch.
learn.fit_one_cycle(5)
learn.recorder.plot_losses()
learn.recorder.plot_lr()
learn.recorder.plot_lr(show_moms=True)
Note that metrics are only collected at the end of each epoch, so you'll need to train at least two epochs to have anything to show here.
learn.recorder.plot_metrics()
Callback methods¶
Inner functions¶
The following functions are used along the way by the Recorder
or can be called by other callbacks.
Module functions¶
Note that you have to create the Optimizer
yourself if you call this function, whereas Learn.fit
creates it for you automatically.
You won't generally need to call this yourself - it's what fit
calls for each epoch.
This is what fit
calls after each epoch. You can call it if you want to run inference on a DataLoader
manually.