FAVDBench
Dataset
FAVDBench (Fine-grained Audible Video Description Benchmark) is a fine-grained audio-visual description benchmark dataset proposed at CVPR 2023, aimed at providing detailed textual descriptions for audible videos that include object appearance, spatial location, action information, and sound descriptions.
Dataset Highlights
A fine-grained description benchmark for audio-visual understanding, pushing the frontier of multimodal research
Audio-Visual Fusion
Covering both visual and auditory information, it is one of the few datasets that incorporates audio descriptions into video description benchmarks, supporting cross-modal research.
Fine-Grained Descriptions
Provides fine-grained text annotations across five dimensions: Appearance, Spatial, Temporal, Action, and Audio.
Multimodal Annotations
Each video contains multi-dimensional human annotation information, with high annotation quality, suitable for training and evaluating multimodal generation models.
Academic Benchmark
Proposed and established by a CVPR 2023 paper, widely cited in academia, it is the standard evaluation benchmark for audio-visual description tasks.
Diverse Content
The videos cover a variety of scenes and themes, including natural scenes, human activities, animal behaviors, etc., ensuring a comprehensive assessment of model generalization capabilities.
Openly Available
The dataset is released under an open license, allowing researchers to freely download and use it, lowering the barriers for academic research and industrial applications.
Applicable Scenarios
Empowering audio and video understanding technology from academic research to industrial applications
Video Description Generation
Train and evaluate video description generation models to automatically generate multidimensional natural language descriptions for videos
Audio and Video Understanding
Research the joint understanding of visual and auditory information, exploring cross-modal semantic alignment and fusion methods
Multimodal Research
Provide high-quality training and evaluation data for visual-language-audio three-modal pre-trained models
Video Subtitle Generation
Develop automatic video subtitle systems to enhance the accessibility and retrievability of video content
Data Preview
The following are annotation examples from the FAVDBench dataset, containing refined descriptions across five dimensions
{
"video_id": "video_00123",
"descriptions": {
"appearance": "A brown dog with floppy ears and a red collar stands on green grass.",
"spatial": "The dog is positioned in the center of the frame with trees in the background.",
"temporal": "The video starts with the dog sitting, then it stands up and begins to walk.",
"action": "The dog wags its tail, barks twice, and runs toward the camera.",
"audio": "Birds chirping in the background, followed by two loud barks and rustling grass."
},
"duration": 8.5,
"split": "train"
}
3 Steps to Get Started Quickly
From browsing to usage, you can start your multimodal research in just a few minutes
Browse Datasets
View dataset details on the Ace Data Cloud platform to understand metadata such as annotation format, data scale, and usage licenses.
Download Data
Download video files and JSON annotation data; the dataset provides standard splits for training, validation, and testing sets.
Load and Use
Use json.load() to load annotation data and start training and evaluating multimodal models with video processing libraries.
Start Exploring the FAVDBench Dataset
CVPR 2023 Fine-grained Audio-Video Description Benchmark, open license, download now. Whether you are a multimodal researcher or a video understanding engineer, this dataset is worth a try.