WanJuan 2.0
Dataset
WanJuan2.0 (WanJuan-CC) is a high-quality English web text dataset of 1T Tokens obtained from CommonCrawl. Released by the Shanghai Artificial Intelligence Laboratory, it has undergone a rigorous quality filtering pipeline, including URL filtering, text extraction, deduplication, quality assessment, and more.
Dataset Highlights
Wanjun 2.0 is a carefully designed quality filtering pipeline that provides high-quality English corpus for pre-training large language models.
Trillion-scale Corpus
Contains high-quality English text with 1 trillion tokens, providing ample data support for pre-training large-scale language models to meet the training needs of GPT-level models.
Strict Quality Pipeline
Undergoes multi-stage refined processing including URL filtering, text extraction, content deduplication, and quality scoring to ensure the high quality and usability of each data point.
CommonCrawl Source
Extracted from CommonCrawl, the world's largest open web crawling archive, covering various types of web pages including news, encyclopedias, academic articles, and blogs.
Open Source Apache-2.0
Uses a permissive Apache-2.0 open source license, allowing commercial use, modification, and redistribution, lowering the barrier to use.
LLM Pre-training Optimization
The dataset is specifically designed and optimized for pre-training scenarios of large language models, with targeted data cleaning and quality screening, directly compatible with mainstream training frameworks.
Comprehensive Documentation
Accompanied by detailed methodology papers and data processing flow descriptions, clearly documenting the strategies and parameters of each filtering stage for easy reproduction and secondary research.
Applicable Scenarios
From large model pre-training to data quality research, covering various cutting-edge NLP research directions
LLM Pre-training
Providing trillion-level English pre-training corpus for large language models like GPT and LLaMA, supporting large-scale language model training from scratch
Text Quality Research
Studying the effectiveness of web text quality filtering strategies, analyzing the impact of different filtering methods on downstream model performance
NLP Benchmark Testing
Using large-scale English corpus to build and evaluate NLP task benchmarks, measuring model performance on various natural language processing tasks
Data Curation Research
Studying the impact of data quality on model performance, exploring optimal data selection and ratio strategies, guiding pre-training dataset construction
Data Preview
The following is a JSONL format example of the WanJuan-CC dataset, each record contains text content, source URL, and quality score
{"text": "The development of large language models has transformed natural language processing...", "url": "https://example.com/article1", "quality_score": 0.92}
{"text": "Recent advances in deep learning architectures have enabled...", "url": "https://example.com/article2", "quality_score": 0.88}
{"text": "Machine learning applications in healthcare continue to expand...", "url": "https://example.com/article3", "quality_score": 0.95}
3 Steps to Get Started Quickly
From browsing to training, you can start your large model pre-training project in just a few minutes
Browse Dataset
View the details of the WanJuan-CC dataset on the Ace Data Cloud platform, including metadata such as data scale, format description, and licensing agreements.
Download Data Shards
Obtain data files through the download methods provided by the platform, supporting streaming or bulk downloads to accommodate different storage and bandwidth conditions.
Tokenization and Training
Perform Tokenize processing on the downloaded text data, and you can directly input it into your large language model training process to start pre-training experiments.
Start Exploring WanJuan-CC Data
Trillions of high-quality English corpus, Apache-2.0 open source license, available immediately. Whether you are a large language model researcher or a pre-training engineer, WanJuan-CC is an ideal choice for training data.