Using Datasets, Pipeline, TfmdLists and Transform in text
from fastai.basics import *
from fastai.callback.all import *
from fastai.text.all import *

In this tutorial, we explore the mid-level API for data collection in the text application. We will use the bases introduced in the pets tutorial so you should be familiar with Transform, Pipeline, TfmdLists and Datasets already.

## Data

path = untar_data(URLs.WIKITEXT_TINY)

The dataset comes with the articles in two csv files, so we read it and concatenate them in one dataframe.

df_all = pd.concat([df_train, df_valid])
0
0 \n = 2013 – 14 York City F.C. season = \n \n The 2013 – 14 season was the <unk> season of competitive association football and 77th season in the Football League played by York City Football Club , a professional football club based in York , North Yorkshire , England . Their 17th @[email protected] place finish in 2012 – 13 meant it was their second consecutive season in League Two . The season ran from 1 July 2013 to 30 June 2014 . \n Nigel Worthington , starting his first full season as York manager , made eight permanent summer signings . By the turn of the year York were only above the relegation z...
1 \n = Big Boy ( song ) = \n \n " Big Boy " <unk> " I 'm A Big Boy Now " was the first single ever recorded by the Jackson 5 , which was released by Steeltown Records in January 1968 . The group played instruments on many of their Steeltown compositions , including " Big Boy " . The song was neither a critical nor commercial success , but the Jackson family were delighted with the outcome nonetheless . \n The Jackson 5 would release a second single with Steeltown Records before moving to Motown Records . The group 's recordings at Steeltown Records were thought to be lost , but they were re...
2 \n = The Remix ( Lady Gaga album ) = \n \n The Remix is a remix album by American recording artist Lady Gaga . Released in Japan on March 3 , 2010 , it contains remixes of the songs from her first studio album , The Fame ( 2008 ) , and her third extended play , The Fame Monster ( 2009 ) . A revised version of the track list was prepared for release in additional markets , beginning with Mexico on May 3 , 2010 . A number of recording artists have produced the songs , including Pet Shop Boys , Passion Pit and The Sound of Arrows . The remixed versions feature both uptempo and <unk> composit...
3 \n = New Year 's Eve ( Up All Night ) = \n \n " New Year 's Eve " is the twelfth episode of the first season of the American comedy television series Up All Night . The episode originally aired on NBC in the United States on January 12 , 2012 . It was written by Erica <unk> and was directed by Beth McCarthy @[email protected] Miller . The episode also featured a guest appearance from Jason Lee as Chris and Reagan 's neighbor and Ava 's boyfriend , Kevin . \n During Reagan ( Christina Applegate ) and Chris 's ( Will <unk> ) first New Year 's Eve game night , Reagan 's competitiveness comes out causing Ch...
4 \n = Geopyxis carbonaria = \n \n Geopyxis carbonaria is a species of fungus in the genus Geopyxis , family <unk> . First described to science in 1805 , and given its current name in 1889 , the species is commonly known as the charcoal loving elf @[email protected] cup , dwarf <unk> cup , <unk> <unk> cup , or pixie cup . The small , <unk> @[email protected] shaped fruitbodies of the fungus are reddish @[email protected] brown with a whitish fringe and measure up to 2 cm ( 0 @[email protected] 8 in ) across . They have a short , tapered stalk . Fruitbodies are commonly found on soil where brush has recently been burned , sometimes in great numbers ....
df_train.iloc[-1].values
dtype=object)

We could tokenize it based on spaces to compare (as is usually done) but here we'll use the standard fastai tokenizer.

splits = [list(range_of(df_train)), list(range(len(df_train), len(df_all)))]
tfms = [attrgetter("text"), Tokenizer.from_df(0), Numericalize()]
dsets = Datasets(df_all, [tfms], splits=splits, dl_type=LMDataLoader)
bs,sl = 104,72
dls.show_batch()
text text_
0 xxbos = xxmaj plunketts xxmaj creek ( xxmaj loyalsock xxmaj creek ) = \n▁\n▁ xxmaj plunketts xxmaj creek is an approximately xxunk - long ( 10.0 km ) tributary of xxmaj loyalsock xxmaj creek in xxmaj lycoming and xxmaj sullivan counties in the xxup u.s . state of xxmaj pennsylvania . xxmaj two unincorporated villages and a hamlet are on the creek , and its watershed drains 23.6 square miles ( 61 = xxmaj plunketts xxmaj creek ( xxmaj loyalsock xxmaj creek ) = \n▁\n▁ xxmaj plunketts xxmaj creek is an approximately xxunk - long ( 10.0 km ) tributary of xxmaj loyalsock xxmaj creek in xxmaj lycoming and xxmaj sullivan counties in the xxup u.s . state of xxmaj pennsylvania . xxmaj two unincorporated villages and a hamlet are on the creek , and its watershed drains 23.6 square miles ( 61 km2
1 xxmaj the early stories follow the pattern of children 's books of orthodox education , such as those by xxmaj heinrich xxmaj hoffmann 's xxunk , that aim to teach the devastating consequences of bad behaviour . xxmaj busch did not assign value to his work , as he once explained to xxmaj heinrich xxmaj richter : " i look at my things for what they are , as xxunk xxunk [ the early stories follow the pattern of children 's books of orthodox education , such as those by xxmaj heinrich xxmaj hoffmann 's xxunk , that aim to teach the devastating consequences of bad behaviour . xxmaj busch did not assign value to his work , as he once explained to xxmaj heinrich xxmaj richter : " i look at my things for what they are , as xxunk xxunk [ toys
2 for some xxmaj egyptologists , such as xxunk xxunk , this failure contributed in no small part to the fall of the xxmaj old xxmaj kingdom , but others , including xxmaj strudwick , believe the reasons of the collapse must be sought elsewhere as the power of an administration official never approached that of the king . \n▁ xxmaj the reforms of xxmaj djedkare xxmaj isesi played an important role in some xxmaj egyptologists , such as xxunk xxunk , this failure contributed in no small part to the fall of the xxmaj old xxmaj kingdom , but others , including xxmaj strudwick , believe the reasons of the collapse must be sought elsewhere as the power of an administration official never approached that of the king . \n▁ xxmaj the reforms of xxmaj djedkare xxmaj isesi played an important role in flourishing
3 aspect of the sport – they are the most basic level of track and field competition . xxmaj meetings are generally organised annually either under the patronage of an educational institution or sports club , or by a group or business that serves as the meeting promoter . xxmaj in the case of the former , athletes are selected to represent their club or institution . xxmaj in the case of privately of the sport – they are the most basic level of track and field competition . xxmaj meetings are generally organised annually either under the patronage of an educational institution or sports club , or by a group or business that serves as the meeting promoter . xxmaj in the case of the former , athletes are selected to represent their club or institution . xxmaj in the case of privately run
4 by oracles communicating the gods ' will . \n▁\n▁ = = = xxmaj worship = = = \n▁\n▁ xxmaj official religious practices , which maintained maat for the benefit of all xxmaj egypt , were related to , but distinct from , the religious practices of ordinary people , who sought the gods ' help for their personal problems . \n▁ xxmaj official religion involved a variety of rituals , based in oracles communicating the gods ' will . \n▁\n▁ = = = xxmaj worship = = = \n▁\n▁ xxmaj official religious practices , which maintained maat for the benefit of all xxmaj egypt , were related to , but distinct from , the religious practices of ordinary people , who sought the gods ' help for their personal problems . \n▁ xxmaj official religion involved a variety of rituals , based in temples
5 xxmaj rachel as one of the earliest and most prominent examples of the xxmaj jewish xxmaj american xxmaj princess stereotype on screen . xxmaj writing for the xxmaj university of xxmaj north xxmaj carolina at xxmaj chapel xxmaj hill , xxmaj alicia xxup r. xxunk also acknowledged xxmaj rachel 's initial xxmaj jewish xxmaj american xxmaj princess qualities , describing her as " spoiled , dependent on her father 's money and rachel as one of the earliest and most prominent examples of the xxmaj jewish xxmaj american xxmaj princess stereotype on screen . xxmaj writing for the xxmaj university of xxmaj north xxmaj carolina at xxmaj chapel xxmaj hill , xxmaj alicia xxup r. xxunk also acknowledged xxmaj rachel 's initial xxmaj jewish xxmaj american xxmaj princess qualities , describing her as " spoiled , dependent on her father 's money and her
6 continues to attempt to xxunk . xxmaj the pair then encounter a xxmaj canadian who gives them his xxunk . \n▁ xxmaj continuing north , they soon run out of gas , but receive help from the xxmaj aurora xxmaj boreanaz , who instructs them to stay at a nearby cabin . xxmaj the two survive the night in the cabin and set out on foot the next morning . xxmaj they to attempt to xxunk . xxmaj the pair then encounter a xxmaj canadian who gives them his xxunk . \n▁ xxmaj continuing north , they soon run out of gas , but receive help from the xxmaj aurora xxmaj boreanaz , who instructs them to stay at a nearby cabin . xxmaj the two survive the night in the cabin and set out on foot the next morning . xxmaj they finally
7 xxmaj general 's world as a labyrinth is xxunk by his constant return to cities and towns he has visited before : each location belongs to the past as well as to the present . xxmaj the xxmaj general in his xxmaj labyrinth xxunk the lines between xxunk in a man - made world and wandering in the natural world . \n▁\n▁ = = = xxmaj fate and love = = = general 's world as a labyrinth is xxunk by his constant return to cities and towns he has visited before : each location belongs to the past as well as to the present . xxmaj the xxmaj general in his xxmaj labyrinth xxunk the lines between xxunk in a man - made world and wandering in the natural world . \n▁\n▁ = = = xxmaj fate and love = = = \n▁\n▁
8 direct measurement of the distance to a star ( 61 xxunk at 11.4 light - years ) was made in 1838 by xxmaj friedrich xxunk using the parallax technique . xxunk measurements demonstrated the vast separation of the stars in the heavens . xxmaj observation of double stars gained increasing importance during the 19th century . xxmaj in 1834 , xxmaj friedrich xxunk observed changes in the proper motion of the star measurement of the distance to a star ( 61 xxunk at 11.4 light - years ) was made in 1838 by xxmaj friedrich xxunk using the parallax technique . xxunk measurements demonstrated the vast separation of the stars in the heavens . xxmaj observation of double stars gained increasing importance during the 19th century . xxmaj in 1834 , xxmaj friedrich xxunk observed changes in the proper motion of the star xxmaj

## Model

config = awd_lstm_lm_config.copy()
config.update({'input_p': 0.6, 'output_p': 0.4, 'weight_p': 0.5, 'embed_p': 0.1, 'hidden_p': 0.2})
model = get_language_model(AWD_LSTM, len(dls.vocab), config=config)