Data Callbacks

Callbacks which work with a learner’s data

CollectDataCallback

 CollectDataCallback (after_create=None, before_fit=None,
                      before_epoch=None, before_train=None,
                      before_batch=None, after_pred=None, after_loss=None,
                      before_backward=None, after_cancel_backward=None,
                      after_backward=None, before_step=None,
                      after_cancel_step=None, after_step=None,
                      after_cancel_batch=None, after_batch=None,
                      after_cancel_train=None, after_train=None,
                      before_validate=None, after_cancel_validate=None,
                      after_validate=None, after_cancel_epoch=None,
                      after_epoch=None, after_cancel_fit=None,
                      after_fit=None)

Collect all batches, along with pred and loss, into self.data. Mainly for testing

source

WeightedDL

 WeightedDL (dataset=None, bs=None, wgts=None, shuffle:bool=False,
             num_workers:int=None, verbose:bool=False, do_setup:bool=True,
             pin_memory=False, timeout=0, batch_size=None,
             drop_last=False, indexed=None, n=None, device=None,
             persistent_workers=False, pin_memory_device='', wif=None,
             before_iter=None, after_item=None, before_batch=None,
             after_batch=None, after_iter=None, create_batches=None,
             create_item=None, create_batch=None, retain=None,
             get_idxs=None, sample=None, shuffle_fn=None, do_batch=None)

Weighted dataloader where wgts is used for the training set only

source

Datasets.weighted_dataloaders

 Datasets.weighted_dataloaders (wgts, bs=64, shuffle_train:bool=None,
                                shuffle:bool=True, val_shuffle:bool=False,
                                n:int=None, path:str|Path='.',
                                dl_type:TfmdDL=None, dl_kwargs:list=None,
                                device:torch.device=None,
                                drop_last:bool=None, val_bs:int=None)

Create a weighted dataloader WeightedDL with wgts for the training set

	Type	Default	Details
wgts
bs	int	64	Batch size
shuffle_train	bool	None	(Deprecated, use `shuffle`) Shuffle training `DataLoader`
shuffle	bool	True	Shuffle training `DataLoader`
val_shuffle	bool	False	Shuffle validation `DataLoader`
n	int	None	Size of `Datasets` used to create `DataLoader`
path	str \| pathlib.Path	.	Path to put in `DataLoaders`
dl_type	TfmdDL	None	Type of `DataLoader`
dl_kwargs	list	None	List of kwargs to pass to individual `DataLoader`s
device	device	None	Device to put `DataLoaders`
drop_last	bool	None	Drop last incomplete batch, defaults to `shuffle`
val_bs	int	None	Validation batch size, defaults to `bs`

lbls = np.random.randint(0, 2, size=(10)) # Dataset of size 10 (train=8, valid=2)
is_valid = lambda i: i >= 8
dblock = DataBlock(blocks=[CategoryBlock], 
    getters=[lambda i: lbls[i]], splitter=FuncSplitter(is_valid))
dset = dblock.datasets(list(range(10)))
item_tfms = [ToTensor()] 
wgts = range(8) # len(wgts) == 8
dls = dset.weighted_dataloaders(bs=1, wgts=wgts, after_item=item_tfms)

dls.show_batch() # if len(wgts) != 8, this will fail"

n = 160
dsets = Datasets(torch.arange(n).float())
dls = dsets.weighted_dataloaders(wgts=range(n), bs=16)
learn = synth_learner(data=dls, cbs=CollectDataCallback)

learn.fit(1)
t = concat(*learn.collect_data.data.itemgot(0,0))
plt.hist(t.numpy());

[0, nan, None, '00:00']

source

DataBlock.weighted_dataloaders

 DataBlock.weighted_dataloaders (source, wgts, bs=64, verbose:bool=False,
                                 shuffle_train:bool=None,
                                 shuffle:bool=True,
                                 val_shuffle:bool=False, n:int=None,
                                 path:str|Path='.', dl_type:TfmdDL=None,
                                 dl_kwargs:list=None,
                                 device:torch.device=None,
                                 drop_last:bool=None, val_bs:int=None)

Create a weighted dataloader WeightedDL with wgts for the dataset

	Type	Default	Details
source
wgts
bs	int	64	Batch size
verbose	bool	False
shuffle_train	bool	None	(Deprecated, use `shuffle`) Shuffle training `DataLoader`
shuffle	bool	True	Shuffle training `DataLoader`
val_shuffle	bool	False	Shuffle validation `DataLoader`
n	int	None	Size of `Datasets` used to create `DataLoader`
path	str \| pathlib.Path	.	Path to put in `DataLoaders`
dl_type	TfmdDL	None	Type of `DataLoader`
dl_kwargs	list	None	List of kwargs to pass to individual `DataLoader`s
device	device	None	Device to put `DataLoaders`
drop_last	bool	None	Drop last incomplete batch, defaults to `shuffle`
val_bs	int	None	Validation batch size, defaults to `bs`

dls = dblock.weighted_dataloaders(list(range(10)), wgts, bs=1)
dls.show_batch()

source

PartialDL

 PartialDL (dataset=None, bs=None, partial_n=None, shuffle:bool=False,
            num_workers:int=None, verbose:bool=False, do_setup:bool=True,
            pin_memory=False, timeout=0, batch_size=None, drop_last=False,
            indexed=None, n=None, device=None, persistent_workers=False,
            pin_memory_device='', wif=None, before_iter=None,
            after_item=None, before_batch=None, after_batch=None,
            after_iter=None, create_batches=None, create_item=None,
            create_batch=None, retain=None, get_idxs=None, sample=None,
            shuffle_fn=None, do_batch=None)

Select randomly partial quantity of data at each epoch

source

FilteredBase.partial_dataloaders

 FilteredBase.partial_dataloaders (partial_n, bs=64,
                                   shuffle_train:bool=None,
                                   shuffle:bool=True,
                                   val_shuffle:bool=False, n:int=None,
                                   path:str|Path='.', dl_type:TfmdDL=None,
                                   dl_kwargs:list=None,
                                   device:torch.device=None,
                                   drop_last:bool=None, val_bs:int=None)

Create a partial dataloader PartialDL for the training set

	Type	Default	Details
partial_n
bs	int	64	Batch size
shuffle_train	bool	None	(Deprecated, use `shuffle`) Shuffle training `DataLoader`
shuffle	bool	True	Shuffle training `DataLoader`
val_shuffle	bool	False	Shuffle validation `DataLoader`
n	int	None	Size of `Datasets` used to create `DataLoader`
path	str \| pathlib.Path	.	Path to put in `DataLoaders`
dl_type	TfmdDL	None	Type of `DataLoader`
dl_kwargs	list	None	List of kwargs to pass to individual `DataLoader`s
device	device	None	Device to put `DataLoaders`
drop_last	bool	None	Drop last incomplete batch, defaults to `shuffle`
val_bs	int	None	Validation batch size, defaults to `bs`

dls = dsets.partial_dataloaders(partial_n=32, bs=16)

assert len(dls[0])==2
for batch in dls[0]:
    assert len(batch[0])==16