# fudan-zvg/reason2drive architecture

> Architecture diagram of the GitHub repository fudan-zvg/reason2drive, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fudan-zvg/reason2drive
- Repository: https://github.com/fudan-zvg/reason2drive

## Overview

##### Project type and purpose
- Type: Research-oriented data‐processing and benchmarking framework for autonomous driving.
- Purpose: Construct a large video-text dataset with chain-based reasoning annotations (perception, prediction, reasoning) and demonstrate a multi-stage inference pipeline.

##### “file structure”
- README.md: Contains all project description, annotation schema, methodology, dataset stats.
- img/: Holds illustrative figures (annotation schema, dataset distribution, pipeline).

##### Key insights from README
- External sources: nuScenes, Waymo, ONCE
- Annotation stages: parsing raw metadata → JSON entries → template injection → GPT-4 + manual verification
- Data output: mini/full Reason2Drive benchmark (video‐text pairs, long-form reasoning)
- Inference pipeline: Perception → Prediction → Chain-based Reasoning → Visualization

##### Main components and their interactions
a. Data Ingestion & Parsing
- External Datasets (nuScenes, Waymo, ONCE)
- Metadata Parser → JSON entries
b. Annotation & Enrichment
- Template Engine (per perception, prediction, reasoning tasks)
- GPT-4 Auto-Annotation
- Manual QA & Enrichment
c. Benchmark Dataset
- Structured Video-Text Pairs
- Task-specific splits (perception/prediction/reasoning)
d. Model Pipeline (Reason2Drive Framework)
- Perception Module (e.g., object detection models via nuScenes-devkit, LAVIS)
- Prediction Module (motion forecasting)
- Reasoning Module (chain-of-thought using MiniGPT-4 or similar)
- Visualization Component (overlay predicted location/motion)
e. External Libraries & Tools
- nuScenes-devkit, LAVIS, MiniGPT-4

##### Architectural patterns & design principles
- Separation of Concerns: distinct data-processing, annotation, dataset management, and inference stages.
- Modular Pipeline: each stage (perception, prediction, reasoning) is a pluggable component.
- Extensibility: templates and annotation stages can be extended to new tasks or data sources.
- Reproducibility: use of standard dataset tools and clear JSON-structured metadata.

##### Putting it all together
- Top layer: External Data Sources → Parser → JSON Database
- Middle layer: Template Injector → GPT-4 & Manual QA → Final Dataset (with splits)
- Bottom layer: Reason2Drive Inference Pipeline (Perception → Prediction → Reasoning → Visualization)
- Side annotations: list key frameworks (nuScenes-devkit, LAVIS, MiniGPT-4) next to respective modules.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources & Ingestion
    subgraph "Data Sources & Ingestion"
        direction TB
        nuScenes["nuScenes"]:::datasrc
        Waymo["Waymo"]:::datasrc
        ONCE["ONCE"]:::datasrc
        MetadataParser["Metadata Parser"]:::process
        JSONStore["JSON Store"]:::storage
    end

    %% Annotation & Dataset Construction
    subgraph "Annotation & Dataset Construction"
        direction TB
        TemplateEngine["Template Engine"]:::process
        GPT4["GPT-4 Auto-Annotation"]:::process
        ManualQA["Manual QA & Enrichment"]:::manual
        FinalDataset["Benchmark Dataset\n(Video-Text Pairs & Splits)"]:::dataset
    end

    %% Inference Pipeline
    subgraph "Reason2Drive Inference Pipeline"
        direction TB
        Perception["Perception Module"]:::pipeline
        Prediction["Prediction Module"]:::pipeline
        Reasoning["Chain-based Reasoning Module"]:::pipeline
        Visualization["Visualization Component"]:::pipeline
    end

    %% External Tools & Libraries
    subgraph "External Tools & Libraries"
        direction TB
        NuScenesDevkit["nuScenes-devkit"]:::tool
        LAVIS["LAVIS"]:::tool
        MiniGPT4["MiniGPT-4"]:::tool
    end

    %% Connections
    nuScenes -->|"Raw sensor logs"| MetadataParser
    Waymo -->|"Raw sensor logs"| MetadataParser
    ONCE -->|"Raw sensor logs"| MetadataParser
    MetadataParser -->|"JSON entries"| JSONStore
    JSONStore --> TemplateEngine
    TemplateEngine --> GPT4
    GPT4 -.-> ManualQA
    ManualQA --> FinalDataset
    TemplateEngine --> FinalDataset
    FinalDataset --> Perception
    Perception --> Prediction
    Prediction --> Reasoning
    Reasoning --> Visualization

    %% Tools associations
    NuScenesDevkit --> Perception
    LAVIS --> Perception
    MiniGPT4 --> Reasoning

    %% Click Events
    click README "https://github.com/fudan-zvg/reason2drive/blob/main/README.md"
    click Figures "https://github.com/fudan-zvg/reason2drive/tree/main/img/"
    click MetadataParser "https://github.com/fudan-zvg/reason2drive/blob/main/README.md"
    click TemplateEngine "https://github.com/fudan-zvg/reason2drive/blob/main/README.md"
    click GPT4 "https://github.com/fudan-zvg/reason2drive/blob/main/README.md"
    click ManualQA "https://github.com/fudan-zvg/reason2drive/blob/main/README.md"
    click FinalDataset "https://github.com/fudan-zvg/reason2drive/blob/main/README.md"
    click Perception "https://github.com/fudan-zvg/reason2drive/blob/main/README.md"
    click Prediction "https://github.com/fudan-zvg/reason2drive/blob/main/README.md"
    click Reasoning "https://github.com/fudan-zvg/reason2drive/blob/main/README.md"
    click Visualization "https://github.com/fudan-zvg/reason2drive/blob/main/README.md"
    click NuScenesDevkit "https://github.com/fudan-zvg/reason2drive/blob/main/README.md"
    click LAVIS "https://github.com/fudan-zvg/reason2drive/blob/main/README.md"
    click MiniGPT4 "https://github.com/fudan-zvg/reason2drive/blob/main/README.md"

    %% Styles
    classDef datasrc fill:#AED6F1,stroke:#1F618D,color:#1B2631;
    classDef process fill:#F9E79F,stroke:#B9770E,color:#1B2631;
    classDef storage fill:#D5F5E3,stroke:#1D8348,color:#1B2631;
    classDef dataset fill:#FADBD8,stroke:#922B21,color:#1B2631;
    classDef pipeline fill:#D6EAF8,stroke:#2E86C1,color:#1B2631;
    classDef tool fill:#F5CBA7,stroke:#B9770E,color:#1B2631;
    classDef manual fill:none,stroke:#E74C3C,stroke-dasharray: 5 5;color:#1B2631;
```
