FineWeb
Ultra-large-scale pre-trained dataset
FineWeb is an ultra-large-scale high-quality English web text dataset released by Hugging Face, containing 15 trillion tokens, selected from 96 Common Crawl snapshots, specifically designed for pre-training large language models. It employs a multi-stage data quality filtering pipeline (URL filtering, text extraction, deduplication, model quality scoring), surpassing similar datasets like C4, Dolma, and RedPajama. ```
Dataset Highlights
An industrial-grade high-quality corpus designed for pre-training large language models
Ultra-large scale
Contains 15 trillion tokens, making it the largest open English web text dataset, providing ample corpus for training GPT/LLaMA level large language models.
Quality-first pipeline
Utilizes a multi-stage data cleaning process: URL filtering, trafilatura text extraction, MinHash deduplication, model quality scoring, ensuring high quality for each data point.
Surpassing peer datasets
In LLM pre-training benchmark tests, FineWeb's performance significantly surpasses mainstream open-source datasets like C4, Dolma, and RedPajama.
CommonCrawl source
Extracted from 96 complete Common Crawl snapshots from 2013 to 2024, covering a wide range of web content and time spans.
Open license (ODC-By 1.0)
Utilizes the Open Data Commons Attribution license, allowing commercial use, modification, and redistribution with proper attribution.
Hugging Face ecosystem
Seamlessly integrates with the Hugging Face datasets library, supporting data streaming without needing to download all data to start training.
Applicable Scenarios
Comprehensively supports the LLM development process from model training to academic research
LLM pre-training
Train GPT, LLaMA level large language models from scratch, providing a vast amount of high-quality English corpus support
Continuous pre-training
Continuously pre-train existing foundational models for domain adaptation, enhancing language understanding capabilities in specific fields
Data quality research
Study the design and effects of data filtering pipelines, exploring the impact of data quality scoring on model performance
Benchmarking and evaluation
Compare the impact of different dataset qualities on model performance, providing experimental basis for dataset selection
Data Preview
The following is a JSONL format example of the FineWeb dataset, with each line containing text content, source URL, token count, and quality score
{"text": "Artificial intelligence continues to reshape industries across the globe...", "url": "https://example.com/ai-article", "token_count": 2847, "quality_score": 0.94}
{"text": "The latest advances in renewable energy technology promise...", "url": "https://example.com/energy-tech", "token_count": 1523, "quality_score": 0.91} {"text": "Understanding neural network architectures requires...", "url": "https://example.com/nn-guide", "token_count": 4102, "quality_score": 0.97}
3 Steps to Get Started Quickly
From browsing to training, quickly integrate FineWeb into your LLM development process
Browse Datasets
Browse the FineWeb datasets on the Ace Data Cloud platform, and view details such as metadata, licensing agreements, and data scale.
Stream Load or Download
Stream load data partitions via the Hugging Face datasets library, or download specific CC snapshot subsets on demand.
Integrate into Training Pipeline
Feed the data into your tokenizer and training pipeline, supporting mainstream deep learning frameworks such as PyTorch, JAX, and TensorFlow.
Start Exploring FineWeb Data
150 trillion tokens of high-quality English corpus, open license, available immediately. Whether you are training the next generation of large language models or researching data quality pipelines, FineWeb is your ideal choice.