This is a bundle of three text data sets to be used for NLP research.

• Dialog system technology challenge 7 (DSTC7)

• Ubuntu

• Advising

• Wikitext-103

An implementation of a transformer network using this data can be found here.

Content

The data set features a preprocessed version and a raw version of the three data sets.    
In data.7z there are three folders for three different NLP tasks:

• CL for classification on the DSTC dataset

• LM-DSTC for building a language model on the DSTC dataset

• and LM-WIKI103 also for building a language model but on the wikitext-103 data set.    
  The .npy files can be loaded by using numpys np.load() function and the .pkl files can be loaded using pythons pickle module.

The Test, Train and Validation files contain the Ubuntu and Advising data sets and wikitext-103-raw, the wikitext-103 data set.

Acknowledgements

I did not create these datasets and I also didn't develop any of the methods used for preprocessing them. I simply took the time to organize them into the format here and did the preprocessing.