Dolma Open
Corpus
Dolma is a large-scale open corpus created by Allen AI, containing 3 trillion tokens, integrating six major data sources: Common Crawl, The Stack, C4, Reddit, Wikipedia, and Semantic Scholar, for training the OLMo series language models, and is currently one of the most transparent large-scale pre-training datasets.
Dataset Highlights
A large-scale, multi-source, fully transparent open pre-training corpus
Trillions of Tokens
Contains approximately 30 trillion tokens of text data, making it one of the largest publicly available pre-training corpora, providing ample data support for training large language models.
Six Major Data Sources
Integrates six major sources: Common Crawl web pages, The Stack code, C4 filtered text, Reddit conversations, Wikipedia encyclopedia, and Semantic Scholar academic papers.
Fully Transparent
Allen AI has made the complete data collection, cleaning, deduplication, and filtering processes public, with each processing step being traceable and auditable, setting a new benchmark for dataset transparency.
Quality Filtering Pipeline
Employs a multi-level quality filtering pipeline, including language detection, content filtering, deduplication, and toxicity detection steps, ensuring the overall quality of the training data.
Reproducible Processing
All data processing code is open-sourced on GitHub, allowing researchers to fully reproduce the entire processing flow from raw data to the final corpus.
Open License
Uses the ODC-By 1.0 open data license, allowing free use for academic research and commercial applications, with proper attribution required.
Applicable Scenarios
Empowering the AI community from model training to data science research
LLM Pre-training
Providing diverse and large-scale text data as core pre-training corpus for training foundational models from scratch
Data Ratio Research
Exploring the optimal mixing ratios of different data sources, studying the impact of web pages, code, encyclopedias, academic papers, etc. on model capabilities
Ablation Experiments
Systematically studying the independent contributions of each data component to model performance by removing or replacing specific data sources
Reproducible AI Research
Ensuring that research results are verifiable and reproducible based on completely open data and processing workflows, promoting scientific rigor in the AI field
API Call Example
Quickly obtain Dolma dataset information through the Ace Data Cloud API
import requests
url = "https://api.acedata.cloud/datasets/dolma"
headers = {
"Authorization": "Bearer YOUR_API_TOKEN",
"Accept": "application/json"
}
response = requests.get(url, headers=headers)
data = response.json()
# View basic information of the dataset
print(f"Name: {data['name']}")
print(f"Token Count: {data['tokens']}")
print(f"Data Sources: {data['sources']}")
print(f"License: {data['license']}")
3 Steps to Get Started Quickly
From understanding to usage, quickly start your journey with large model training data
Browse the Dataset
View the details of the Dolma dataset on the Ace Data Cloud platform, understand the composition of data sources, token scale, and licensing agreements.
Get API Token
Register and obtain your API Token to access the dataset through the api.acedata.cloud interface.
Download and Train
Download the required data shards via API and start your pre-training or research experiments with the Dolma corpus for your large language model.
Start Exploring the Dolma Open Corpus
30 trillion tokens, 6 major data sources, completely transparent processing flow. Whether you are training the next generation of language models or conducting data science research, Dolma is the ideal choice.