LMDrive
Dataset
The LMDrive dataset consists of 64,000 instruction-sensor-control data segments collected in the CARLA simulator, each containing navigation instructions, multimodal multi-view sensor data, and control signals, lasting from 2 to 20 seconds, suitable for language-driven autonomous driving research. ```
Dataset Highlights
A large-scale multimodal benchmark dataset for language-driven autonomous driving
Language Navigation Instructions
Each data segment comes with natural language navigation instructions, such as "Turn right at the next intersection," supporting research on the alignment of language and driving behavior.
Multi-view Camera Data
Includes sequences of RGB camera images from front, left, right, and other views, providing rich visual perception information for scene understanding.
LiDAR Point Cloud
Each frame includes high-precision LiDAR 3D point cloud data, which can be used for obstacle detection, depth estimation, and 3D spatial perception tasks.
Control Signals
Records precise control signals such as steering angle, throttle, and brake, supporting the training and evaluation of end-to-end driving models.
CARLA Simulator Collection
Data is collected in the high-fidelity CARLA simulator, covering various weather, lighting, and traffic scenarios, safely reproducing various driving situations.
Diverse Driving Scenarios
Covers various road types and traffic scenarios, including urban roads, intersections, roundabouts, and highways, enhancing model generalization capabilities.
Applicable Scenarios
From academic research to industrial prototype validation, covering core directions of autonomous driving
Language-Guided Driving
Train driving models that can understand natural language navigation instructions, achieving human-machine language interactive autonomous driving.
End-to-End Autonomous Driving
Direct end-to-end learning from sensor input to control output, exploring driving strategies without intermediate representations.
Multimodal Fusion
Integrate various modal data such as camera images, LiDAR point clouds, and language instructions to enhance perception and decision-making capabilities.
Instruction Following
Evaluate the model's understanding and execution capabilities of human navigation instructions, promoting the development of interactive autonomous driving systems.
Data Preview
The following is a typical structure of a single data segment, including instructions, sensor paths, and control signals
{
"instruction": "Turn right at the next intersection.",
"segment_id": "route_00042_seg_0015",
"duration": 8.2,
"sensors": {
"camera_front": "data/route_00042/seg_0015/rgb_front/*.png",
"camera_left": "data/route_00042/seg_0015/rgb_left/*.png",
"camera_right": "data/route_00042/seg_0015/rgb_right/*.png",
"lidar": "data/route_00042/seg_0015/lidar/*.npy"
},
"control": [
{ "frame": 0, "steering": 0.00, "throttle": 0.60, "brake": 0.00 }, { "frame": 10, "steering": 0.35, "throttle": 0.45, "brake": 0.00 }, { "frame": 20, "steering": 0.72, "throttle": 0.30, "brake": 0.00 },
{ "frame": 30, "steering": 0.50, "throttle": 0.55, "brake": 0.00 },
{ "frame": 40, "steering": 0.05, "throttle": 0.65, "brake": 0.00 }
],
"weather": "ClearNoon",
"town": "Town03"
}
3 Steps to Get Started Quickly
From browsing to loading, you can start your autonomous driving research in just a few minutes
Browse the Dataset
View the details of the LMDrive dataset on the Ace Data Cloud platform to understand metadata such as data structure, scene distribution, and licensing agreements.
Download Data
Download 64,000 command-sensor-control data segments, including multi-angle camera images, LiDAR point clouds, and control signals.
Load and Train
Use the official LMDrive toolkit to load the data and begin training and evaluating the language-driven autonomous driving model.
Start Exploring the LMDrive Dataset
A large-scale language-driven autonomous driving dataset with open licensing, available for immediate download. Whether you are an autonomous driving researcher or a multimodal AI developer, this dataset is worth trying.