This dataset is a synthetic dataset of short stories generated by GPT-3.5 and GPT-4, containing a vocabulary limited to the range of 3 to 4-year-old children's understanding. It is designed for training and evaluating small language models (LMs), and despite the small size of the model (less than 5 million parameters) or simpler architecture (only one transformer block), the model trained using this dataset is still able to produce fluent, consistent, diverse and grammatically perfect short stories.

The TinyStories dataset was proposed by Microsoft Research in 2023, and the relevant paper is “TinyStories: How Small Can Language Models Be and Still Speak Coherent English?"