How taco-group/openemma works
Project Type and Purpose
• Open-source, end-to-end multimodal ML pipeline for autonomous driving
• Integrates vision language models (GPT-4, LLaVA, Qwen, Llama, etc.) with YOLO-3D object detection
• Performs data ingestion (nuScenes), multimodal encoding, motion-planning head, and visualization/serving
File Structure
• Top-level modules:
– llava/: Core model wrappers, multimodal encoder/projector, training scripts, serving (CLI, Gradio)
– openemma/YOLO3D/: Standalone 3D object detection tool (Dockerfile, model configs, inference scripts)
– openemma/vlm/: Base backbones and VLM integration
– main.py: Orchestration entrypoint for inference/training
• Architectural patterns: modular layering (core model vs. external detector vs. serve vs. train), plugin-style model builders
Key Insights from README
– Waymo EMMA (concept)
– OpenEMMA pipeline (our implementation)
• Pipeline stages:
1. Data Loader (nuScenes dataset)
2. YOLO-3D Inference (critical object detection)
3. VLM Reasoning (front-view images + text → waypoint predictions + rationales via OpenAI GPT-4 or local LLaVA)
4. Visualization & Output (annotated images, videos, waypoint logs)
• External dependencies: OpenAI API, pretrained VLMs, PyTorch + CUDA, Docker for YOLO3D
Main Components & Relationships
A. Data Ingestion
– Component: nuScenes Dataset
B. YOLO-3D Detector (openemma/YOLO3D)
– Inputs: raw front-view frames
C. Multimodal Encoder (llava/mm_utils, llava/model/multimodal_encoder)
– Merges image features + object metadata + textual prompts
– Uses builder patterns: clip_encoder, projector, language_model wrappers
D. Motion Planning Head (llava/model/llava_arch, llava/model/language_model)
– Predicts future waypoints + decision rationales
– Interacts with external API (OpenAI GPT-4) or local models
– Annotates frames, compiles videos
F. Serving Layer (llava/serve)
– CLI, Gradio web server, model_worker, controller
– Exposes REST / interactive demo interfaces
Separately include Training Pipeline:
– llava/train/** (train.py, llava_trainer.py, flash_attn / xformers monkey patches)
Architectural Patterns & Technologies
• Modular plugin architecture: builder.py factories for swapping language backbones (Llama, Mistral, MPT)
• Layered ML pipeline: Detector → Encoder → Planner → Visualizer/Serve
• External service integration: OpenAI REST API for GPT-4 reasoning
• Containerization: Dockerfiles in YOLO3D & GAE utilities
• Frameworks: PyTorch, CUDA (flash-attn, xformers), HuggingFace transformers, Gradio, Flask REST API