# opendrivelab/openpilot-deepdive architecture

> Architecture diagram of the GitHub repository opendrivelab/openpilot-deepdive, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/opendrivelab/openpilot-deepdive
- Repository: https://github.com/opendrivelab/openpilot-deepdive

## Overview

##### Project Type and Purpose
- Type: A deep learning experimental framework for autonomous driving.
- Purpose: To reimplement and evaluate an end-to-end neural network (the “Supercombo” model) that predicts vehicle trajectories directly from camera inputs. It supports training, testing (with distributed training capabilities on clusters or bare-metal machines), and demo visualization.

##### Analyzing the File Structure and Key Components
- Core Scripts and Modules:
- main.py: The central driver for training and evaluation.
- model.py: Contains the architecture of the neural network.
- data.py and utils.py: Likely include data loading, preprocessing, and helper functions.
- demo.py: Utility for running inference/demonstrations and writing visualization outputs.
- view_transform.py: Handles camera model transformations.
- Tools:
- The tools/ directory includes scripts (e.g., extract_comma2k19.py, extract_nuscenes.py) for dataset-specific preprocessing and splitting.
- Utilities Specific to Comma2k19:
- The utils_comma2k19 package mirrors utilities from the official commaai repository, containing modules for handling cameras, coordinates, benchmarks, datasets, and unzipping procedures.
- External and Configuration Files:
- requirements.txt provides dependencies (e.g., PyTorch and likely efficientnet variants), and the presence of SLURM-related instructions in the README denotes support for distributed training.

##### Main System Components
a. Data Ingestion & Preprocessing:
- “Dataset Loader & Preprocessor” block – encompasses data.py, utils.py, and parts of utils_comma2k19 (dataset.py, camera.py, coordinates.py). This block represents how raw data (from Comma2k19 and nuScenes datasets) is ingested, preprocessed (including frame extraction), and prepared for training.
- “Dataset Split Tools” block – representing the tools/ directory scripts for creating dataset splits.
- “Data Storage” – the external datasets (soft-linked under the data folder). Indicate that this is user provided and that the system expects them to be correctly linked.

b. Model and Training Pipeline:
- “Model Architecture” block – represented by model.py. Mention that the pretrained weights (e.g., efficientnet-b2) are loaded and used as part of the network.
- “Training Controller” block – depicted by main.py, which ties together data feeding, training logic, and evaluation procedure.
- “Distributed Training Manager” block – conceptual block showing how the training process supports both cluster (with SLURM and environment variables) and bare-metal multi-GPU setups.

c. Inference / Demo Pipeline:
- “Demo & Visualization” block – representing demo.py and the visualization output (e.g., frames saved in the “vis” folder and further conversion to video via ffmpeg). Also include view_transform.py here to indicate image coordinate transformation.
- This block receives outputs from the trained model and supports real-time or offline scenario generation.

d. Auxiliary Components:
- “Utility Functions and Modules” block – such as utils_comma2k19 that provide standardized functionalities (benchmarking, coordinate transformations, etc.).
- “Logging and Monitoring” block – mention the integration with tensorboard for logging loss/metric curves.
- “Configuration & Environment Setup” block – touch on requirements.txt and settings for batch size, number of workers, and environmental variables for distributed training.

##### Architectural Patterns and Principles
- End-to-End Pipeline: The design follows a linear, end-to-end flow where data ingestion, preprocessing, model inference/training, and demo visualization are interconnected.
- Modularity: Separation into distinct modules (data loading, preprocessing, model definition, demo generation) that can be represented as discrete blocks.
- Distributed Processing: An explicit block for distributed training highlighting the use of SLURM and coordinated multi-GPU setups.
- Reusability: The utils_comma2k19 namespace indicates library-like reuse of functions originally provided by commaai.

- Annotations indicating which parts are influenced by external settings (batch size, SLURM variables, etc.), and possibly add comments on memory-intensive operations (as highlighted in the README).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion & Preprocessing Subgraph
    subgraph "Data Ingestion & Preprocessing"
        D1["Data Loader (data.py)"]:::preprocessing
        U1["Utils (utils.py)"]:::preprocessing
        subgraph "Comma2k19 Utilities"
            C1["Dataset (utils_comma2k19/dataset.py)"]:::preprocessing
            C2["Camera (utils_comma2k19/camera.py)"]:::preprocessing
            C3["Coordinates (utils_comma2k19/coordinates.py)"]:::preprocessing
        end
    end

    %% Dataset Split Tools Subgraph
    subgraph "Dataset Split Tools"
        T1["Extract Comma2k19 (tools/extract_comma2k19.py)"]:::utilities
        T2["Extract nuScenes (tools/extract_nuscenes.py)"]:::utilities
    end

    %% Data Storage (External Datasets)
    DS["Data Storage (User Provided Datasets)"]:::storage

    %% Model & Training Pipeline Subgraph
    subgraph "Model & Training Pipeline"
        M1["Model Architecture (model.py)"]:::training
        TC["Training Controller (main.py)"]:::training
        DTM["Distributed Training Manager"]:::training
    end

    %% Inference & Demo Pipeline Subgraph
    subgraph "Inference & Demo Pipeline"
        DMO["Demo & Visualization (demo.py)"]:::demo
        VT["Camera Model Transformation (view_transform.py)"]:::demo
    end

    %% Other Supporting Components
    LM["Logging & Monitoring (TensorBoard)"]:::training
    CFG["Configuration & Environment Setup (requirements.txt)"]:::config

    %% Data Flow Connections
    DS -->|"ingests"| D1
    DS -->|"ingests"| U1
    T1 -->|"split_data"| D1
    T2 -->|"split_data"| D1
    C1 -->|"assists"| D1
    C2 -->|"assists"| D1
    C3 -->|"assists"| D1
    D1 -->|"preprocessed_data"| M1
    U1 -->|"preprocessed_data"| M1

    %% Training Pipeline Connections
    M1 -->|"feeds_model"| TC
    DTM -->|"controls"| TC
    M1 -->|"logs"| LM
    TC -->|"reports"| LM
    CFG -->|"configures"| TC
    CFG -->|"configures"| M1

    %% Inference & Demo Connections
    M1 -->|"inference_output"| DMO
    VT -->|"transforms"| DMO

    %% Click Events for Component Mapping
    click TC "https://github.com/opendrivelab/openpilot-deepdive/blob/main/main.py"
    click M1 "https://github.com/opendrivelab/openpilot-deepdive/blob/main/model.py"
    click D1 "https://github.com/opendrivelab/openpilot-deepdive/blob/main/data.py"
    click U1 "https://github.com/opendrivelab/openpilot-deepdive/blob/main/utils.py"
    click C1 "https://github.com/opendrivelab/openpilot-deepdive/blob/main/utils_comma2k19/dataset.py"
    click C2 "https://github.com/opendrivelab/openpilot-deepdive/blob/main/utils_comma2k19/camera.py"
    click C3 "https://github.com/opendrivelab/openpilot-deepdive/blob/main/utils_comma2k19/coordinates.py"
    click T1 "https://github.com/opendrivelab/openpilot-deepdive/blob/main/tools/extract_comma2k19.py"
    click T2 "https://github.com/opendrivelab/openpilot-deepdive/blob/main/tools/extract_nuscenes.py"
    click DMO "https://github.com/opendrivelab/openpilot-deepdive/blob/main/demo.py"
    click VT "https://github.com/opendrivelab/openpilot-deepdive/blob/main/view_transform.py"
    click CFG "https://github.com/opendrivelab/openpilot-deepdive/blob/main/requirements.txt"

    %% Styles
    classDef preprocessing fill:#ffefd5,stroke:#333,stroke-width:2px;
    classDef utilities fill:#e6e6fa,stroke:#333,stroke-width:2px;
    classDef training fill:#add8e6,stroke:#333,stroke-width:2px;
    classDef demo fill:#98fb98,stroke:#333,stroke-width:2px;
    classDef storage fill:#ffe4e1,stroke:#333,stroke-width:2px;
    classDef config fill:#fffacd,stroke:#333,stroke-width:2px;
```
