Wandb

Integration with Weights & Biases

First thing first, you need to install wandb with

pip install wandb

Create a free account then run

wandb login

in your terminal. Follow the link to get an API token that you will need to paste, then you’re all set!

WandbCallback sends the model topology, losses, metrics, hyperparameters, and prediction samples to a wandb run, plus the dataset and the model checkpoint when asked. The run must exist before training starts:

before_fit checks that wandb.init has run, logs the learner’s config through gather_args, watches the model once per process, logs the dataset when asked, and prepares the fixed sample of validation items whose predictions are logged:

source

WandbCallback.before_fit

def before_fit():

Call watch method to log model topology, gradients & weights

Each training batch logs the losses, the hyperparameters, and the throughput:

source

WandbCallback.after_batch

def after_batch():

Log hyper-parameters and training loss

source

WandbCallback.before_batch

def before_batch():

log_predictions fetches the predictions for the held sample and hands them to wandb_process for display:

source

WandbCallback.log_predictions

def log_predictions():

Each epoch logs the validation metrics, and the predictions when log_preds_every_epoch:

source

WandbCallback.after_epoch

def after_epoch():

Log validation loss and custom metrics & log prediction samples

After training the predictions are logged once more when they were not logged per epoch, and the saved checkpoint is uploaded when log_model:

source

WandbCallback.after_fit

def after_fit():

source

WandbCallback

def WandbCallback(
    log:str=None, # What to log (can be `gradients`, `parameters`, `all` or None)
    log_preds:bool=True, # Whether to log model predictions on a `wandb.Table`
    log_preds_every_epoch:bool=False, # Whether to log predictions every epoch or at the end
    log_model:bool=False, # Whether to save the model checkpoint to a `wandb.Artifact`
    model_name:str=None, # The name of the `model_name` to save, overrides `SaveModelCallback`
    log_dataset:bool=False, # Whether to log the dataset to a `wandb.Artifact`
    dataset_name:str=None, # A name to log the dataset with
    valid_dl:fastai.data.core.TfmdDL=None, # If `log_preds=True`, then the samples will be drawn from `valid_dl`
    n_preds:int=36, # How many samples to log predictions
    seed:int=12345, # The seed of the samples drawn
    reorder:bool=True
):

Saves model topology, losses & metrics

Optionally logs weights and or gradients depending on log (can be “gradients”, “parameters”, “all” or None), sample predictions if log_preds=True that will come from valid_dl or a random sample of the validation set (determined by seed). n_preds are logged in this case.

If used in combination with SaveModelCallback, the best model is saved as well (can be deactivated with log_model=False).

Datasets can also be tracked:

For custom scenarios, you can also manually use functions log_dataset and log_model to respectively log your own datasets and models.

source

Learner.gather_args

def gather_args():

Gather config parameters accessible to the learner

source

Learner.gather_args

def gather_args():

Gather config parameters accessible to the learner

Config values are logged recursively. Dicts and lists are walked, and an object with constructor arguments is logged as those arguments plus its name:

Artifact metadata is logged as strings:

log_dataset uploads a folder as a dataset artifact, skipping its models subfolder:

source

log_dataset

def log_dataset(
    path, name:NoneType=None, metadata:dict={}, description:str='raw dataset'
):

Log dataset folder

log_model uploads a saved checkpoint as a model artifact:

source

log_model

def log_model(
    path, name:NoneType=None, metadata:dict={}, description:str='trained model'
):

Log model file

wandb_process turns a batch of samples and predictions into what wandb displays for them, with a @dispatch case per input and target type. Images with any target overlay the prediction and the truth on the input:

Image classification is a table of image, label, and prediction:

Segmentation overlays the truth and predicted masks on the image:

Text classification is a table of text, target, and prediction:

Example of use:

Once your have defined your Learner, before you call to fit or fit_one_cycle, you need to initialize wandb:

import wandb
wandb.init()

To use Weights & Biases without an account, you can call wandb.init(anonymous='allow').

Then you add the callback to your learner or call to fit methods, potentially with SaveModelCallback if you want to save the best model:

from fastai.callback.wandb import *

# To log only during one training phase
learn.fit(..., cbs=WandbCallback())

# To log continuously for all training phases
learn = learner(..., cbs=WandbCallback())

Datasets and models can be tracked through the callback or directly through log_model and log_dataset functions.

For more details, refer to W&B documentation.