Falcon RefinedWeb
Dataset
A high-quality English web dataset meticulously constructed by the Technology Innovation Institute (TII) in the UAE, containing 968 million records and 600 billion tokens, is the core pre-training data for the Falcon series of large language models, with extreme quality filtering applied to Common Crawl through the MaDR pipeline.
Dataset Highlights
Industrial-grade web data cleaning pipeline, tailored for pre-training large language models
Web-scale
Built on Common Crawl, containing 968 million records and 600 billion tokens, large enough to support the pre-training of large language models from scratch.
MaDR Cleaning Pipeline
Utilizes Macroscopic Data Refinement pipeline, ensuring data quality through content extraction, quality filtering, and deduplication.
Extreme Deduplication
Employs MinHash + LSH for large-scale fuzzy deduplication, combined with precise substring deduplication, effectively eliminating redundant content and near-duplicate documents.
URL Filtering
Filters malicious, pornographic, and low-quality domains through blacklists and heuristic rules, controlling the safety and reliability of the dataset from the source.
Language Recognition
Uses fastText language classifier to accurately filter non-English content, ensuring high language purity of the dataset, suitable for English model training.
Open for Commercial Use
Released under ODC-By 1.0 license, allowing free use and redistribution, including commercial purposes, with attribution required.
Applicable Scenarios
From basic model training to data quality research, covering various NLP applications
LLM Pre-training
As the core corpus for pre-training large language models from scratch, validated on models such as Falcon-7B/40B/180B
Data Quality Research
Researching the impact of web data cleaning and filtering strategies on model performance, comparing the effects of different quality pipelines
Filtering Benchmark Testing
As a benchmark dataset for data filtering and deduplication algorithms, evaluating and optimizing the performance of data cleaning pipelines
Basic Model Training
Providing high-quality English corpus for custom basic models, supporting various architectures and training paradigms
API Call Example
Quickly obtain Falcon RefinedWeb data through the Ace Data Cloud API
curl -X GET "https://api.acedata.cloud/datasets/falcon-refinedweb" \
-H "Authorization: Bearer YOUR_API_TOKEN" \
-H "Content-Type: application/json"
# Response Example
{
"content": "The history of artificial intelligence began in antiquity, with myths, stories and rumors of artificial beings endowed with intelligence...",
"url": "https://en.wikipedia.org/wiki/History_of_artificial_intelligence",
"timestamp": "2023-06-15T08:22:31Z",
"token_count": 1247
}
3 Steps to Get Started Quickly
From browsing to obtaining data, you can start your model training project in just a few minutes
Browse the Dataset
View the details of the Falcon RefinedWeb dataset on the Ace Data Cloud platform to understand metadata such as scale, fields, and licensing agreements.
Obtain API Token
Register and create an API Token to access data on demand via the RESTful API, without needing to download the entire dataset.
Load and Train
Use the data obtained via the API directly for model pre-training, data analysis, or quality research, ready to use out of the box.
Start Exploring Falcon RefinedWeb Data
Industry-grade web pre-training corpus with open commercial licensing, available via API immediately. Whether you are training a base model or researching data quality, this dataset is an ideal choice.