How fudan-zvg/reason2drive works
Project type and purpose
• Type: Research-oriented data‐processing and benchmarking framework for autonomous driving.
• Purpose: Construct a large video-text dataset with chain-based reasoning annotations (perception, prediction, reasoning) and demonstrate a multi-stage inference pipeline.
“file structure”
• README.md: Contains all project description, annotation schema, methodology, dataset stats.
• img/: Holds illustrative figures (annotation schema, dataset distribution, pipeline).
Key insights from README
• External sources: nuScenes, Waymo, ONCE
• Annotation stages: parsing raw metadata → JSON entries → template injection → GPT-4 + manual verification
• Data output: mini/full Reason2Drive benchmark (video‐text pairs, long-form reasoning)
• Inference pipeline: Perception → Prediction → Chain-based Reasoning → Visualization
Main components and their interactions
a. Data Ingestion & Parsing
• External Datasets (nuScenes, Waymo, ONCE)
• Metadata Parser → JSON entries
b. Annotation & Enrichment
• Template Engine (per perception, prediction, reasoning tasks)
• GPT-4 Auto-Annotation
• Manual QA & Enrichment
c. Benchmark Dataset
• Structured Video-Text Pairs
• Task-specific splits (perception/prediction/reasoning)
d. Model Pipeline (Reason2Drive Framework)
• Perception Module (e.g., object detection models via nuScenes-devkit, LAVIS)
• Prediction Module (motion forecasting)
• Reasoning Module (chain-of-thought using MiniGPT-4 or similar)
• Visualization Component (overlay predicted location/motion)
e. External Libraries & Tools
• nuScenes-devkit, LAVIS, MiniGPT-4
Architectural patterns & design principles
• Separation of Concerns: distinct data-processing, annotation, dataset management, and inference stages.
• Modular Pipeline: each stage (perception, prediction, reasoning) is a pluggable component.
• Extensibility: templates and annotation stages can be extended to new tasks or data sources.
• Reproducibility: use of standard dataset tools and clear JSON-structured metadata.
Putting it all together
• Top layer: External Data Sources → Parser → JSON Database
• Middle layer: Template Injector → GPT-4 & Manual QA → Final Dataset (with splits)
• Bottom layer: Reason2Drive Inference Pipeline (Perception → Prediction → Reasoning → Visualization)
• Side annotations: list key frameworks (nuScenes-devkit, LAVIS, MiniGPT-4) next to respective modules.