# taco-group/openemma architecture

> Architecture diagram of the GitHub repository taco-group/openemma, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/taco-group/openemma
- Repository: https://github.com/taco-group/openemma

## Overview

##### Project Type and Purpose
- Open-source, end-to-end **multimodal ML pipeline** for autonomous driving
- Integrates vision language models (GPT-4, LLaVA, Qwen, Llama, etc.) with YOLO-3D object detection
- Performs data ingestion (nuScenes), multimodal encoding, motion-planning head, and visualization/serving

##### File Structure
- Top-level modules:
– **llava/**: Core model wrappers, multimodal encoder/projector, training scripts, serving (CLI, Gradio)
– **openemma/YOLO3D/**: Standalone 3D object detection tool (Dockerfile, model configs, inference scripts)
– **openemma/vlm/**: Base backbones and VLM integration
– **main.py**: Orchestration entrypoint for inference/training
- Architectural patterns: modular layering (core model vs. external detector vs. serve vs. train), plugin-style model builders

##### Key Insights from README
– Waymo EMMA (concept)
– OpenEMMA pipeline (our implementation)
- Pipeline stages:
1. **Data Loader** (nuScenes dataset)
2. **YOLO-3D Inference** (critical object detection)
3. **VLM Reasoning** (front-view images + text → waypoint predictions + rationales via OpenAI GPT-4 or local LLaVA)
4. **Visualization & Output** (annotated images, videos, waypoint logs)
- External dependencies: OpenAI API, pretrained VLMs, PyTorch + CUDA, Docker for YOLO3D

##### Main Components & Relationships
A. Data Ingestion
– Component: nuScenes Dataset
B. YOLO-3D Detector (openemma/YOLO3D)
– Inputs: raw front-view frames
C. Multimodal Encoder (llava/mm_utils, llava/model/multimodal_encoder)
– Merges image features + object metadata + textual prompts
– Uses builder patterns: clip_encoder, projector, language_model wrappers
D. Motion Planning Head (llava/model/llava_arch, llava/model/language_model)
– Predicts future waypoints + decision rationales
– Interacts with external API (OpenAI GPT-4) or local models
– Annotates frames, compiles videos
F. Serving Layer (llava/serve)
– CLI, Gradio web server, model_worker, controller
– Exposes REST / interactive demo interfaces

Separately include **Training Pipeline**:
– llava/train/** (train.py, llava_trainer.py, flash_attn / xformers monkey patches)

##### Architectural Patterns & Technologies
- **Modular plugin architecture**: builder.py factories for swapping language backbones (Llama, Mistral, MPT)
- **Layered ML pipeline**: Detector → Encoder → Planner → Visualizer/Serve
- **External service integration**: OpenAI REST API for GPT-4 reasoning
- **Containerization**: Dockerfiles in YOLO3D & GAE utilities
- **Frameworks**: PyTorch, CUDA (flash-attn, xformers), HuggingFace transformers, Gradio, Flask REST API

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        DataIngestion["nuScenes Dataset"]:::storage
    end

    %% Processing Layer
    subgraph "Processing Layer"
        direction TB
        subgraph "YOLO-3D Detector Service"
            direction TB
            Y3D_Service["YOLO-3D Detector"]:::compute
            Y3D_Infer["inference.py"]:::compute
            Y3D_Main["main.py"]:::compute
            Y3D_Models["models/"]:::compute
            Y3D_Utils["utils/"]:::compute
            Y3D_Docker["Dockerfile"]:::compute
        end
        subgraph "Multimodal Encoder"
            direction TB
            MM_Encoder["Multimodal Encoder"]:::compute
            MM_Builder["builder.py"]:::compute
            MM_Clip["clip_encoder.py"]:::compute
            MM_Projector["multimodal_projector/"]:::compute
        end
        subgraph "Motion Planning Head"
            direction TB
            Planner["Motion Planning Head"]:::compute
            Planner_Arch["llava_arch.py"]:::compute
            LM_Wrappers["language_model/"]:::compute
        end
    end

    %% External Services
    OpenAI["OpenAI GPT-4 API"]:::external

    %% Interface Layer
    subgraph "Interface Layer"
        direction TB
        Visualize["Visualization Module"]:::interface
        Viz_Script["visualize.py"]:::interface
        Viz_Samples["samples/output.json"]:::interface
        subgraph "Serving Layer"
            direction TB
            CLI["CLI Interface"]:::interface
            Gradio["Gradio Web Server"]:::interface
            Controller["controller.py"]:::interface
            ModelWorker["model_worker.py"]:::interface
            SGLang["sglang_worker.py"]:::interface
        end
    end

    %% Training Layer
    subgraph "Training Layer"
        direction TB
        Train["Training Pipeline"]:::training
        Train_Script["train.py"]:::training
        Trainer["llava_trainer.py"]:::training
        Patch1["llama_flash_attn_monkey_patch.py"]:::training
        Patch2["llama_xformers_attn_monkey_patch.py"]:::training
        AltTrain1["train_mem.py"]:::training
        AltTrain2["train_xformers.py"]:::training
    end

    %% Utilities & Constants
    subgraph "Core Utilities"
        direction TB
        UtilsShared["utils.py"]:::compute
        Constants["constants.py"]:::compute
        ModelFactory["model/builder.py"]:::compute
    end

    %% Orchestration
    Orchestrator["main.py"]:::compute

    %% Relationships
    DataIngestion -->|images| Y3D_Service
    Y3D_Service -->|bboxes & metadata| MM_Encoder
    MM_Encoder -->|features + text prompt| Planner
    Planner -->|waypoints + rationale| Visualize
    Planner -->|waypoints + rationale| CLI
    Planner -->|waypoints + rationale| Gradio
    Planner -->|API calls| OpenAI
    DataIngestion -->|optional aug| Train_Script
    Train_Script -->|augmented data| Y3D_Service
    Train_Script -->|training features| MM_Encoder
    Train_Script -->|training loop| Trainer
    Trainer -->|checkpoint| Constants

    Orchestrator -->|run inference/train| Y3D_Service
    Orchestrator --> MM_Encoder
    Orchestrator --> Planner
    Orchestrator --> Visualize
    Orchestrator --> CLI

    %% Click Events
    click DataIngestion "https://github.com/taco-group/openemma/blob/main/openemma/YOLO3D/utils/datasets.py"
    click Y3D_Service "https://github.com/taco-group/openemma/tree/main/openemma/YOLO3D/"
    click Y3D_Infer "https://github.com/taco-group/openemma/blob/main/openemma/YOLO3D/inference.py"
    click Y3D_Main "https://github.com/taco-group/openemma/blob/main/openemma/YOLO3D/main.py"
    click Y3D_Models "https://github.com/taco-group/openemma/tree/main/openemma/YOLO3D/models/"
    click Y3D_Utils "https://github.com/taco-group/openemma/tree/main/openemma/YOLO3D/utils/"
    click Y3D_Docker "https://github.com/taco-group/openemma/tree/main/openemma/YOLO3D/Dockerfile"
    click MM_Builder "https://github.com/taco-group/openemma/blob/main/llava/model/multimodal_encoder/builder.py"
    click MM_Clip "https://github.com/taco-group/openemma/blob/main/llava/model/multimodal_encoder/clip_encoder.py"
    click MM_Projector "https://github.com/taco-group/openemma/tree/main/llava/model/multimodal_projector/"
    click Planner_Arch "https://github.com/taco-group/openemma/blob/main/llava/model/llava_arch.py"
    click LM_Wrappers "https://github.com/taco-group/openemma/tree/main/llava/model/language_model/"
    click Orchestrator "https://github.com/taco-group/openemma/blob/main/main.py"
    click Train_Script "https://github.com/taco-group/openemma/blob/main/llava/train/train.py"
    click Trainer "https://github.com/taco-group/openemma/blob/main/llava/train/llava_trainer.py"
    click Patch1 "https://github.com/taco-group/openemma/blob/main/llava/train/llama_flash_attn_monkey_patch.py"
    click Patch2 "https://github.com/taco-group/openemma/blob/main/llava/train/llama_xformers_attn_monkey_patch.py"
    click AltTrain1 "https://github.com/taco-group/openemma/blob/main/llava/train/train_mem.py"
    click AltTrain2 "https://github.com/taco-group/openemma/blob/main/llava/train/train_xformers.py"
    click UtilsShared "https://github.com/taco-group/openemma/blob/main/llava/utils.py"
    click Constants "https://github.com/taco-group/openemma/blob/main/llava/constants.py"
    click ModelFactory "https://github.com/taco-group/openemma/blob/main/llava/model/builder.py"
    click Visualize "https://github.com/taco-group/openemma/blob/main/openemma/visualize/visualize.py"
    click Viz_Samples "https://github.com/taco-group/openemma/blob/main/openemma/visualize/samples/output.json"
    click CLI "https://github.com/taco-group/openemma/blob/main/llava/serve/cli.py"
    click Gradio "https://github.com/taco-group/openemma/blob/main/llava/serve/gradio_web_server.py"
    click Controller "https://github.com/taco-group/openemma/blob/main/llava/serve/controller.py"
    click ModelWorker "https://github.com/taco-group/openemma/blob/main/llava/serve/model_worker.py"
    click SGLang "https://github.com/taco-group/openemma/blob/main/llava/serve/sglang_worker.py"
    click BaseBB "https://github.com/taco-group/openemma/blob/main/openemma/vlm/base_backbone.py"

    %% Styles
    classDef storage fill:#ADD8E6,stroke:#000;
    classDef compute fill:#98FB98,stroke:#000;
    classDef external fill:#FFA500,stroke:#000;
    classDef interface fill:#DDA0DD,stroke:#000;
    classDef training fill:#FFFACD,stroke:#000;
```
