OmniCorpus-CW
Dataset
OmniCorpus-CW contains 1 billion image-text interleaved documents selected from the OmniCorpus-CC dataset, sourced from Common Crawl web scraping data, and is currently one of the largest open-source image-text interleaved datasets, suitable for multimodal large language model pre-training.
Dataset Highlights
Ultra-large-scale image-text mixed corpus provides a solid foundation for pre-training multimodal large models
Billion-level Corpus
Includes 1 billion filtered image-text mixed documents, making it one of the largest open-source multimodal mixed datasets, providing ample data support for large-scale pre-training.
Image-Text Mixed Format
Text and images in the documents are naturally interleaved, preserving the original image-text association structure of the web pages, closer to the real-world presentation of multimodal information.
Diversity of Web Crawling
The data is sourced from Common Crawl's large-scale web crawling, covering various types of web pages such as news, encyclopedias, blogs, and forums, with a wide distribution across fields.
Selected Quality Filtering
Based on OmniCorpus-CC, it has undergone a strict quality screening process to remove low-quality and duplicate content, ensuring the usability and training effectiveness of the corpus.
Multi-domain Coverage
Covers web content from numerous fields such as technology, education, life, and business, helping models learn a wide range of world knowledge and visual-semantic alignment capabilities.
LLM Pre-training Ready
The data format and scale are specifically designed for pre-training multimodal large language models and can be directly used in the training processes of models like Flamingo and OpenFlamingo.
Applicable Scenarios
From multimodal pre-training to web content analysis, adaptable to various research directions
Multimodal LLM Pre-training
Provide large-scale image-text interleaved pre-training corpus for multimodal large models like Flamingo and Emu, enhancing the model's visual-language understanding ability
Visual Language Research
Research cutting-edge topics such as cross-modal alignment, reference resolution, and context understanding in image-text interleaved scenarios
Document Understanding
Train models to understand web documents containing mixed layouts of images and text, enhancing document-level multimodal understanding ability
Web Content Analysis
Analyze the distribution patterns of images and text, content themes, and information structures in large-scale web pages for web data mining research
Data Preview
The following is a structural example of image-text interleaved documents, each document contains an alternating sequence of text paragraphs and image URLs
{
"id": "omnicorpus-cw-000001",
"source": "common_crawl",
"content": [
{"type": "text", "value": "The solar system consists of the Sun and the celestial objects bound to it by gravity."},
{"type": "image", "url": "https://example.com/images/solar_system.jpg", "alt": "Solar System Diagram"},
{"type": "text", "value": "The four inner planets — Mercury, Venus, Earth, and Mars — are terrestrial planets."},
{"type": "image", "url": "https://example.com/images/inner_planets.jpg", "alt": "Inner Planets"},
{"type": "text", "value": "Beyond the asteroid belt lie the gas giants Jupiter and Saturn, and the ice giants Uranus and Neptune."}
],
"metadata": {
"url": "https://example.com/science/solar-system",
"language": "en",
"num_images": 2,
"num_text_segments": 3
}
}
3 Steps to Get Started Quickly
From browsing to loading, you can start your multimodal research project in just a few minutes
Browse Datasets
View dataset details on the Ace Data Cloud platform to understand metadata such as data scale, format description, and licensing agreements.
Download Data
Obtain the OmniCorpus-CW data files through the download link provided by the platform, supporting segmented downloads to accommodate different storage needs.
Load and Train
Use datasets.load_dataset() to load the data and start pre-training or fine-tuning experiments for multimodal models.
Start Exploring the OmniCorpus-CW Dataset
A billion-level image-text interleaved corpus, open license, available immediately. Whether you are a multimodal model researcher or an LLM pre-training engineer, this dataset is an indispensable choice.