# Tensorboard
``` python
from nbdev import show_doc
```
First thing first, you need to install tensorboard with
pip install tensorboard
Then launch tensorboard with
tensorboard --logdir=runs
in your terminal. You can change the logdir as long as it matches the
`log_dir` you pass to
[`TensorBoardCallback`](https://docs.fast.ai/callback.tensorboard.html#tensorboardcallback)
(default is `runs` in the working directory).
## Tensorboard Embedding Projector support
> Tensorboard Embedding Projector is currently only supported for image
> classification
### Export Image Features during Training
Tensorboard [Embedding
Projector](https://www.tensorflow.org/tensorboard/tensorboard_projector_plugin)
is supported in
[`TensorBoardCallback`](https://docs.fast.ai/callback.tensorboard.html#tensorboardcallback)
(set parameter `projector=True`) during training. The validation set
embeddings will be written after each epoch.
cbs = [TensorBoardCallback(projector=True)]
learn = vision_learner(dls, resnet18, metrics=accuracy)
learn.fit_one_cycle(3, cbs=cbs)
### Export Image Features during Inference
To write the embeddings for a custom dataset (e. g. after loading a
learner) use
[`TensorBoardProjectorCallback`](https://docs.fast.ai/callback.tensorboard.html#tensorboardprojectorcallback).
Add the callback manually to the learner.
learn = load_learner('path/to/export.pkl')
learn.add_cb(TensorBoardProjectorCallback())
dl = learn.dls.test_dl(files, with_labels=True)
_ = learn.get_preds(dl=dl)
If using a custom model (non fastai-resnet) pass the layer where the
embeddings should be extracted as a callback-parameter.
layer = learn.model[1][1]
cbs = [TensorBoardProjectorCallback(layer=layer)]
preds = learn.get_preds(dl=dl, cbs=cbs)
### Export Word Embeddings from Language Models
To export word embeddings from Language Models (tested with AWD_LSTM
(fast.ai) and GPT2 / BERT (transformers)) but works with every model
that contains an embedding layer.
For a **fast.ai TextLearner or LMLearner** just pass the learner - the
embedding layer and vocab will be extracted automatically:
dls = TextDataLoaders.from_folder(untar_data(URLs.IMDB), valid='test')
learn = text_classifier_learner(dls, AWD_LSTM, drop_mult=0.5, metrics=accuracy)
projector_word_embeddings(learn=learn, limit=2000, start=2000)
For other language models - like the ones in the **transformers
library** - you’ll have to pass the layer and vocab. Here’s an example
for a **BERT** model.
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")
# get the word embedding layer
layer = model.embeddings.word_embeddings
# get and sort vocab
vocab_dict = tokenizer.get_vocab()
vocab = [k for k, v in sorted(vocab_dict.items(), key=lambda x: x[1])]
# write the embeddings for tb projector
projector_word_embeddings(layer=layer, vocab=vocab, limit=2000, start=2000)
------------------------------------------------------------------------
source
### TensorBoardBaseCallback
``` python
def TensorBoardBaseCallback(
):
```
*Basic class handling tweaks of the training loop by changing a
[`Learner`](https://docs.fast.ai/learner.html#learner) in various
events*
------------------------------------------------------------------------
source
### TensorBoardCallback
``` python
def TensorBoardCallback(
log_dir:NoneType=None, trace_model:bool=True, log_preds:bool=True, n_preds:int=9, projector:bool=False,
layer:NoneType=None
):
```
*Saves model topology, losses & metrics for tensorboard and tensorboard
projector during training*
------------------------------------------------------------------------
source
### TensorBoardProjectorCallback
``` python
def TensorBoardProjectorCallback(
log_dir:NoneType=None, layer:NoneType=None
):
```
*Extracts and exports image featuers for tensorboard projector during
inference*
------------------------------------------------------------------------
source
### projector_word_embeddings
``` python
def projector_word_embeddings(
learn:NoneType=None, layer:NoneType=None, vocab:NoneType=None, limit:int=-1, start:int=0, log_dir:NoneType=None
):
```
*Extracts and exports word embeddings from language models embedding
layers*
## TensorBoardCallback
``` python
from fastai.vision.all import Resize, RandomSubsetSplitter, aug_transforms, vision_learner, resnet18
```
``` python
path = untar_data(URLs.PETS)
db = DataBlock(blocks=(ImageBlock, CategoryBlock),
get_items=get_image_files,
item_tfms=Resize(128),
splitter=RandomSubsetSplitter(train_sz=0.1, valid_sz=0.01),
batch_tfms=aug_transforms(size=64),
get_y=using_attr(RegexLabeller(r'(.+)_\d+.*$'), 'name'))
dls = db.dataloaders(path/'images')
```
``` python
learn = vision_learner(dls, resnet18, metrics=accuracy)
```
``` python
learn.unfreeze()
learn.fit_one_cycle(3, cbs=TensorBoardCallback(Path.home()/'tmp'/'runs'/'tb', trace_model=True))
```
| epoch |
train_loss |
valid_loss |
accuracy |
time |
| 0 |
4.973294 |
5.009670 |
0.082192 |
00:03 |
| 1 |
4.382769 |
4.438282 |
0.095890 |
00:03 |
| 2 |
3.877172 |
3.665855 |
0.178082 |
00:04 |
## Projector
### Projector in TensorBoardCallback
``` python
path = untar_data(URLs.PETS)
```
``` python
db = DataBlock(blocks=(ImageBlock, CategoryBlock),
get_items=get_image_files,
item_tfms=Resize(128),
splitter=RandomSubsetSplitter(train_sz=0.05, valid_sz=0.01),
batch_tfms=aug_transforms(size=64),
get_y=using_attr(RegexLabeller(r'(.+)_\d+.*$'), 'name'))
dls = db.dataloaders(path/'images')
```
``` python
cbs = [TensorBoardCallback(log_dir=Path.home()/'tmp'/'runs'/'vision1', projector=True)]
learn = vision_learner(dls, resnet18, metrics=accuracy)
```
``` python
learn.unfreeze()
learn.fit_one_cycle(3, cbs=cbs)
```
| epoch |
train_loss |
valid_loss |
accuracy |
time |
| 0 |
5.143322 |
6.736727 |
0.082192 |
00:03 |
| 1 |
4.508100 |
5.106580 |
0.109589 |
00:03 |
| 2 |
4.057889 |
4.194602 |
0.068493 |
00:03 |
### TensorBoardProjectorCallback
``` python
path = untar_data(URLs.PETS)
```
``` python
db = DataBlock(blocks=(ImageBlock, CategoryBlock),
get_items=get_image_files,
item_tfms=Resize(128),
splitter=RandomSubsetSplitter(train_sz=0.1, valid_sz=0.01),
batch_tfms=aug_transforms(size=64),
get_y=using_attr(RegexLabeller(r'(.+)_\d+.*$'), 'name'))
dls = db.dataloaders(path/'images')
```
``` python
files = get_image_files(path/'images')
files = files[:256]
```
``` python
dl = learn.dls.test_dl(files, with_labels=True)
```
``` python
learn = vision_learner(dls, resnet18, metrics=accuracy)
layer = learn.model[1][0].ap
cbs = [TensorBoardProjectorCallback(layer=layer, log_dir=Path.home()/'tmp'/'runs'/'vision2')]
```
``` python
_ = learn.get_preds(dl=dl, cbs=cbs)
```
## projector_word_embeddings
### fastai text or lm learner
``` python
from fastai.text.all import TextDataLoaders, text_classifier_learner, AWD_LSTM
```
``` python
dls = TextDataLoaders.from_folder(untar_data(URLs.IMDB), valid='test')
learn = text_classifier_learner(dls, AWD_LSTM, drop_mult=0.5, metrics=accuracy)
```
``` python
projector_word_embeddings(learn, limit=1000, log_dir=Path.home()/'tmp'/'runs'/'text')
```
### transformers
#### GPT2
``` python
from transformers import GPT2LMHeadModel, GPT2TokenizerFast
```
``` python
tokenizer = GPT2TokenizerFast.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')
layer = model.transformer.wte
vocab_dict = tokenizer.get_vocab()
vocab = [k for k, v in sorted(vocab_dict.items(), key=lambda x: x[1])]
projector_word_embeddings(layer=layer, vocab=vocab, limit=2000, log_dir=Path.home()/'tmp'/'runs'/'transformers')
```
#### BERT
``` python
from transformers import AutoTokenizer, AutoModel
```
``` python
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")
layer = model.embeddings.word_embeddings
vocab_dict = tokenizer.get_vocab()
vocab = [k for k, v in sorted(vocab_dict.items(), key=lambda x: x[1])]
projector_word_embeddings(layer=layer, vocab=vocab, limit=2000, start=2000, log_dir=Path.home()/'tmp'/'runs'/'transformers')
```
warning: Embedding dir exists, did you set global_step for add_embedding()?
### Validate results in tensorboard
Run the following command in the command line to check if the projector
embeddings have been correctly wirtten:
tensorboard --logdir=~/tmp/runs
Open http://localhost:6006 in browser (TensorBoard Projector doesn’t
work correctly in Safari!)