TinyStories
Dataset
A dataset of 2.14 million synthetic short stories generated by Microsoft Research using GPT-3.5 and GPT-4, designed for training small language models. It demonstrates that models with only 10 million parameters can generate fluent and coherent English stories. MIT License.
Dataset Highlights
A high-quality synthetic story dataset designed specifically for training small language models
Focus on Small Models
Designed for training small language models (10M-100M parameters), proving that with the right data, small models can generate fluent and coherent English text.
Synthetic High Quality
Generated using GPT-3.5 and GPT-4, each story is guided by carefully designed prompts to ensure narrative quality and linguistic diversity.
Simple Vocabulary
Uses vocabulary and grammatical structures that 3-4 year old children can understand, reducing language complexity to make it easier for small models to learn the basic patterns of language.
Coherent Narratives
Each story contains a complete narrative structure—beginning, development, and ending, helping the model learn the logical reasoning abilities of story-telling.
Diverse Themes
Covers a rich variety of themes such as friendship, adventure, animals, family, and nature, ensuring content diversity through random combinations of keywords and story features.
MIT Open Source
Adopts the MIT license, allowing free commercial and academic use without worrying about licensing restrictions, suitable for various research and application scenarios.
Applicable Scenarios
From cutting-edge research to educational applications, empowering various directions of small language models
Small LM Training
Train small language models with 10M-100M parameters, achieving smooth text generation capabilities under resource-constrained conditions
Language Model Research
Research the Scaling Laws of language models, the impact of data quality on model performance, and the minimum scale threshold for emergent capabilities
Educational AI
Develop AI applications for children's education, such as automatic story generation, reading assistance, and language learning tools
Efficient Model Development
Rapidly iterate model architectures and training strategies on consumer-grade hardware, lowering the computational threshold for NLP research
API Call Example
Quickly access the TinyStories dataset via API
curl -X GET "https://api.acedata.cloud/datasets/tinystories" \
-H "Authorization: Bearer YOUR_API_TOKEN" \
-H "Content-Type: application/json"
# Response
{
"name": "TinyStories",
"description": "2.14M synthetic short stories generated by GPT-3.5/GPT-4",
"license": "MIT",
"source": "Microsoft Research",
"total_stories": 2140000,
"format": "text",
"language": "en"
}
3 Steps to Get Started Quickly
From browsing to training, you can start your small model research in just a few minutes
Browse the Dataset
View the details of the TinyStories dataset on the Ace Data Cloud platform, learn about the data scale, generation method, and licensing agreement.
Download the Data
Obtain 2.14 million synthetic short stories via the API, with a simple data format that is ready to use, requiring no additional cleaning.
Load and Train
Use datasets.load_dataset() to load the data and start training your small language model.
Start Exploring the TinyStories Dataset
2.14 million high-quality synthetic stories, MIT open-source license, available immediately. Whether you are an NLP researcher or an AI education developer, this dataset is an ideal choice for training small language models.