CapsFusion-120M
Image-Text Pair Dataset
A large-scale image-text pair dataset with 120 million images, integrating synthesized descriptions (BLIP2) and web ALT text, generating more accurate and informative image-text descriptions through LLM (ChatGPT). Released by Beijing Academy of AI (BAAI).
Dataset Highlights
Integrating multi-source descriptions to provide high-quality training data for text-to-image and multimodal research
Description Fusion Technology
Fusing synthetic descriptions (generated by BLIP2) and web ALT text through LLM to leverage strengths and compensate for weaknesses, generating high-quality descriptions with both visual accuracy and semantic richness.
Outstanding Description Quality
The fused descriptions significantly outperform the use of original ALT text or purely synthetic descriptions alone, demonstrating stronger training effects across multiple downstream tasks.
120 Million Scale
Contains 120 million image-text pairs, a scale sufficient to support production-level text-to-image models (such as Stable Diffusion, DALL-E) for pre-training and fine-tuning.
LLM Driven Generation
Utilizing ChatGPT as a fusion engine to integrate descriptions from two sources into coherent, natural text, ensuring semantic consistency and readability.
Supported by Academic Papers
Published in the top conference ECCV 2024, accompanied by complete experimental benchmarks and comparative evaluations, with methodology rigorously validated through peer review.
Convenient Integration
Data is stored in Parquet format, compatible with the HuggingFace Datasets library, and can be directly streamed and processed through Python scripts.
Applicable Scenarios
From text-to-image training to multimodal research, covering various cutting-edge applications
Text-to-Image Training
Train text-to-image models like Stable Diffusion and DALL-E using high-quality fused descriptions to enhance the consistency between generated images and text
Image Description Research
Study the impact of different description qualities on downstream tasks, comparing the effects of original descriptions, synthetic descriptions, and fused descriptions
Visual Language Model Training
Provide pre-training data for large-scale Visual Language Models (VLM), enhancing the model's understanding of images and text and cross-modal alignment capabilities
Data Quality Research
Compare and evaluate different description fusion strategies, exploring how to improve model training effectiveness through data engineering
Data Preview
The following are examples of image-text pairs in the dataset, each record contains the original description, synthetic description, and fused description
{"image": "img_00001.jpg", "raw_caption": "dog playing in park", "synthetic_caption": "A golden retriever runs across a grassy field with a red ball", "fused_caption": "A golden retriever enthusiastically chases a red ball across a sunlit park lawn, its ears flapping in the breeze"}
{"image": "img_00002.jpg", "raw_caption": "city skyline at night", "synthetic_caption": "A panoramic view of illuminated skyscrapers reflecting on water", "fused_caption": "A stunning nighttime panorama of a modern city skyline, with illuminated skyscrapers casting shimmering reflections across a calm river"}
3 Steps to Get Started Quickly
From browsing to training, you can integrate high-quality text-image pairs into your workflow in just a few minutes
Browse the Dataset
View the details of the CapsFusion-120M dataset on the Ace Data Cloud platform, understand the field structure, sample size, and licensing agreement.
Download Parquet Shards
Download Parquet format data shard files on demand, supporting batch retrieval, suitable for different storage and computing resources.
Load Fusion Descriptions
Use datasets.load_dataset() to load the data, directly integrating the fusion descriptions into your text-to-image or multimodal training pipeline.
Start Exploring CapsFusion-120M Data
High-quality fusion descriptions at the scale of 120 million, open license, available for immediate download. Whether you are training text-to-image models or exploring multimodal learning, this dataset is an ideal starting point.