# nvlabs/alpamayo2 architecture

> Architecture diagram of the GitHub repository nvlabs/alpamayo2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvlabs/alpamayo2
- Repository: https://github.com/nvlabs/alpamayo2

## Overview

#### Purpose and Entry Points
Alpamayo 2 Super is a Python inference package for a 34B autonomous-vehicle foundation model: a 32B vision-language backbone generates Chain-of-Causation/text-task outputs, and a 2B diffusion action expert samples future ego trajectories. Primary executable entry points are the end-to-end smoke CLI at `src/alpamayo2_super/inference_smoke.py`, its compatibility alias `src/alpamayo2_super/test_inference.py`, notebooks under `notebooks/`, and the explicit two-GPU CFG demo at `examples/two_gpu_nav_cfg_demo.py`.

#### Model Composition and Loading
The core model assembly belongs in `src/alpamayo2_super/models/alpamayo2_super.py`, with the trajectory/action expert implemented by `src/alpamayo2_super/models/expert.py` and action-conditioning projection in `src/alpamayo2_super/models/action_in_proj.py`. Configuration and checkpoint/model-ID handling are centralized in `src/alpamayo2_super/config.py` and `src/alpamayo2_super/common/config_utils.py`. The runtime accepts the gated Hugging Face model `nvidia/Alpamayo2-Super` or a local release-native safetensors checkpoint.

#### AV Input Boundary and Profiles
PhysicalAI-AV dataset retrieval and sample decoding are isolated behind `src/alpamayo2_super/load_physical_aiavdataset.py`. Manifests such as `examples/validation_samples.json` identify clips and timestamps; the loader resolves corresponding camera, ego-history, calibration, and future-trajectory data from Hugging Face’s `nvidia/PhysicalAI-Autonomous-Vehicles` dataset. `src/alpamayo2_super/input_profiles.py` enforces task-specific camera/frame selections, creating a stable boundary between raw dataset assets and model-ready multimodal inputs.

#### Prompting and Text Tasks
Conversation/prompt construction is represented by `src/alpamayo2_super/chat_template/conversation.py`. Task routing and output contracts for reasoning, VQA, grounding, meta-actions, and auto-labeling are concentrated in `src/alpamayo2_super/text_tasks.py`. The backbone’s generated text is both a standalone result and conditioning/context for trajectory generation; downstream consumers should preserve task profile and decoding metadata alongside outputs.

#### Trajectory Sampling Pipeline
The action-space contract and trajectory parameterization live in `src/alpamayo2_super/action_space/action_space.py` and `src/alpamayo2_super/action_space/unicycle_accel_curvature.py`. Diffusion/flow-matching sampling is separated into `src/alpamayo2_super/diffusion/base.py` and `src/alpamayo2_super/diffusion/flow_matching.py`. This stage consumes model conditioning and emits one or more future ego trajectories, which the CLI evaluates against available ground truth (for example, minADE). Geometry transforms used across model/output processing reside in `src/alpamayo2_super/geometry/coordinates.py` and `src/alpamayo2_super/geometry/rotation.py`.

#### Execution, Devices, and Distributed Concerns
Normal CLI/notebook inference uses CUDA PyTorch on Linux and requires enough GPU VRAM for the 34B checkpoint; CUDA 12.x and FlashAttention are material installation/runtime dependencies. Device and distributed primitives are grouped in `src/alpamayo2_super/common/distributed.py`. The advanced CFG path deliberately splits VLM generation onto one GPU and expert denoising plus guided/unguided KV caches onto another, rather than relying on automatic Hugging Face device mapping (`examples/two_gpu_nav_cfg_demo.py`).

#### Outputs and Visualization
Inference produces decoded text, sampled trajectories, metrics, and task/profile metadata; the smoke path serializes JSON plus visualization artifacts. Public plotting APIs are exposed through `src/alpamayo2_super/visualization.py`, with camera projection, six-camera figures, compact layouts, grounding, and task-specific rendering implemented in `src/alpamayo2_super/viz_utils.py`. Calibration-dependent overlays are optional but explicitly report whether projection was available.

Main entry points: [`src/alpamayo2_super/models/alpamayo2_super.py`](https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/models/alpamayo2_super.py) (Alpamayo2 Super model), [`src/alpamayo2_super/inference_smoke.py`](https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/inference_smoke.py) (Smoke inference CLI), [`src/alpamayo2_super/diffusion/flow_matching.py`](https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/diffusion/flow_matching.py) (Flow-matching sampler), [`src/alpamayo2_super/load_physical_aiavdataset.py`](https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/load_physical_aiavdataset.py) (PhysicalAI-AV loader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry points"]
  node_smoke_cli["Smoke inference CLI<br/>Python CLI<br/>[inference_smoke.py]"]
  node_compat_cli["Compatibility CLI alias<br/>Python CLI<br/>[test_inference.py]"]
  node_notebooks["Inference notebooks"]
  node_two_gpu_demo["Two-GPU CFG demo<br/>Python demo"]
end

subgraph group_input["AV input boundary"]
  node_manifest["Validation manifests<br/>Sample manifest"]
  node_physicalai_loader["PhysicalAI-AV loader<br/>Dataset adapter"]
  node_input_profiles["Input profiles<br/>Input policy<br/>[input_profiles.py]"]
end

subgraph group_model["Model runtime"]
  node_conversation["Conversation templates<br/>Prompt builder<br/>[conversation.py]"]
  node_text_tasks["Text task contracts<br/>Task router<br/>[text_tasks.py]"]
  node_config["Model configuration<br/>Config and checkpoint resolver<br/>[config.py]"]
  node_config_utils["Config utilities<br/>Config support<br/>[config_utils.py]"]
  node_distributed["Device and distributed runtime<br/>CUDA runtime<br/>[distributed.py]"]
  node_super_model{{"Alpamayo2 Super model<br/>34B model assembly<br/>[alpamayo2_super.py]"}}
  node_action_projection["Action conditioning projection<br/>Model layer<br/>[action_in_proj.py]"]
  node_action_expert{{"Diffusion action expert<br/>2B trajectory expert<br/>[expert.py]"}}
end

subgraph group_sampling["Trajectory sampling"]
  node_action_space["Action-space contract<br/>Trajectory parameterization<br/>[action_space.py]"]
  node_unicycle["Unicycle accel-curvature space<br/>Action representation"]
  node_diffusion_base["Diffusion base<br/>Sampling framework<br/>[base.py]"]
  node_flow_matching["Flow-matching sampler<br/>Trajectory sampler<br/>[flow_matching.py]"]
  node_geometry["Geometry transforms<br/>Coordinate utilities<br/>[coordinates.py]"]
end

subgraph group_output["Outputs"]
  node_visualization["Visualization API<br/>Plotting facade<br/>[visualization.py]"]
  node_viz_utils["Rendering utilities<br/>Camera and task rendering<br/>[viz_utils.py]"]
  node_results["Inference artifacts<br/>JSON, metrics, and figures"]
end

node_compat_cli -.->|"aliases"| node_smoke_cli
node_manifest -->|"clip references"| node_physicalai_loader
node_physicalai_loader -->|"decoded AV sample"| node_input_profiles
node_input_profiles -->|"model-ready multimodal input"| node_super_model
node_conversation -->|"prompts"| node_super_model
node_text_tasks -->|"task output contract"| node_conversation
node_config_utils -->|"supports"| node_config
node_config -->|"checkpoint and settings"| node_super_model
node_distributed -->|"CUDA placement"| node_super_model
node_super_model -->|"backbone conditioning"| node_action_projection
node_action_projection -->|"projected conditioning"| node_action_expert
node_action_expert -->|"denoising model"| node_flow_matching
node_diffusion_base -->|"sampling base"| node_flow_matching
node_action_space -->|"parameterizes"| node_unicycle
node_unicycle -->|"action representation"| node_flow_matching
node_geometry -->|"trajectory transforms"| node_flow_matching
node_flow_matching -->|"sampled trajectories"| node_results
node_super_model -->|"generated text and metadata"| node_results
node_smoke_cli -->|"loads samples"| node_physicalai_loader
node_smoke_cli -->|"runs inference"| node_super_model
node_smoke_cli -->|"serializes and evaluates"| node_results
node_results -->|"plot inputs"| node_visualization
node_visualization -->|"delegates rendering"| node_viz_utils
node_viz_utils -->|"visualization artifacts"| node_results
node_notebooks -.->|"interactive inference path"| node_smoke_cli
node_two_gpu_demo -->|"split-GPU CFG execution"| node_super_model
node_two_gpu_demo -->|"explicit device control"| node_distributed

click node_smoke_cli "https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/inference_smoke.py"
click node_compat_cli "https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/test_inference.py"
click node_two_gpu_demo "https://github.com/nvlabs/alpamayo2/blob/main/examples/two_gpu_nav_cfg_demo.py"
click node_manifest "https://github.com/nvlabs/alpamayo2/blob/main/examples/validation_samples.json"
click node_physicalai_loader "https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/load_physical_aiavdataset.py"
click node_input_profiles "https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/input_profiles.py"
click node_conversation "https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/chat_template/conversation.py"
click node_text_tasks "https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/text_tasks.py"
click node_config "https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/config.py"
click node_config_utils "https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/common/config_utils.py"
click node_distributed "https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/common/distributed.py"
click node_super_model "https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/models/alpamayo2_super.py"
click node_action_projection "https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/models/action_in_proj.py"
click node_action_expert "https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/models/expert.py"
click node_action_space "https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/action_space/action_space.py"
click node_unicycle "https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/action_space/unicycle_accel_curvature.py"
click node_diffusion_base "https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/diffusion/base.py"
click node_flow_matching "https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/diffusion/flow_matching.py"
click node_geometry "https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/geometry/coordinates.py"
click node_visualization "https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/visualization.py"
click node_viz_utils "https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/viz_utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_smoke_cli,node_compat_cli,node_notebooks,node_two_gpu_demo toneBlue
class node_manifest,node_physicalai_loader,node_input_profiles toneAmber
class node_conversation,node_text_tasks,node_config,node_config_utils,node_distributed,node_super_model,node_action_projection,node_action_expert toneMint
class node_action_space,node_unicycle,node_diffusion_base,node_flow_matching,node_geometry toneRose
class node_visualization,node_viz_utils,node_results toneIndigo
```

## Components

### Entry points

- [Smoke inference CLI](https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/inference_smoke.py): Python CLI · End-to-end inference and evaluation path. · `src/alpamayo2_super/inference_smoke.py`
- [Compatibility CLI alias](https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/test_inference.py): Python CLI · `src/alpamayo2_super/test_inference.py`
- **Inference notebooks**: Notebook
- [Two-GPU CFG demo](https://github.com/nvlabs/alpamayo2/blob/main/examples/two_gpu_nav_cfg_demo.py): Python demo · Places VLM and expert CFG caches on separate GPUs. · `examples/two_gpu_nav_cfg_demo.py`

### AV input boundary

- [Validation manifests](https://github.com/nvlabs/alpamayo2/blob/main/examples/validation_samples.json): Sample manifest · Identifies clips and timestamps. · `examples/validation_samples.json`
- [PhysicalAI-AV loader](https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/load_physical_aiavdataset.py): Dataset adapter · Retrieves and decodes camera, ego, calibration, and future data. · `src/alpamayo2_super/load_physical_aiavdataset.py`
- [Input profiles](https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/input_profiles.py): Input policy · Enforces task-specific camera and frame selection. · `src/alpamayo2_super/input_profiles.py`

### Model runtime

- [Conversation templates](https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/chat_template/conversation.py): Prompt builder · `src/alpamayo2_super/chat_template/conversation.py`
- [Text task contracts](https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/text_tasks.py): Task router · Routes reasoning, VQA, grounding, meta-actions, and auto-labeling. · `src/alpamayo2_super/text_tasks.py`
- [Model configuration](https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/config.py): Config and checkpoint resolver · Resolves Hugging Face model IDs or local safetensors checkpoints. · `src/alpamayo2_super/config.py`
- [Config utilities](https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/common/config_utils.py): Config support · `src/alpamayo2_super/common/config_utils.py`
- [Device and distributed runtime](https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/common/distributed.py): CUDA runtime · Provides device and distributed primitives. · `src/alpamayo2_super/common/distributed.py`
- [Alpamayo2 Super model](https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/models/alpamayo2_super.py): 34B model assembly · Combines the 32B vision-language backbone with the 2B action expert. · `src/alpamayo2_super/models/alpamayo2_super.py`
- [Action conditioning projection](https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/models/action_in_proj.py): Model layer · `src/alpamayo2_super/models/action_in_proj.py`
- [Diffusion action expert](https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/models/expert.py): 2B trajectory expert · `src/alpamayo2_super/models/expert.py`

### Trajectory sampling

- [Action-space contract](https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/action_space/action_space.py): Trajectory parameterization · `src/alpamayo2_super/action_space/action_space.py`
- [Unicycle accel-curvature space](https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/action_space/unicycle_accel_curvature.py): Action representation · `src/alpamayo2_super/action_space/unicycle_accel_curvature.py`
- [Diffusion base](https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/diffusion/base.py): Sampling framework · `src/alpamayo2_super/diffusion/base.py`
- [Flow-matching sampler](https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/diffusion/flow_matching.py): Trajectory sampler · Samples one or more future ego trajectories. · `src/alpamayo2_super/diffusion/flow_matching.py`
- [Geometry transforms](https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/geometry/coordinates.py): Coordinate utilities · `src/alpamayo2_super/geometry/coordinates.py`

### Outputs

- [Visualization API](https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/visualization.py): Plotting facade · `src/alpamayo2_super/visualization.py`
- [Rendering utilities](https://github.com/nvlabs/alpamayo2/blob/main/src/alpamayo2_super/viz_utils.py): Camera and task rendering · Supports calibration-aware overlays and projection availability reporting. · `src/alpamayo2_super/viz_utils.py`
- **Inference artifacts**: JSON, metrics, and figures · Includes text, trajectories, task/profile metadata, and minADE when ground truth exists.

## Connections

- Compatibility CLI alias → Smoke inference CLI: aliases
- Validation manifests → PhysicalAI-AV loader: clip references
- PhysicalAI-AV loader → Input profiles: decoded AV sample
- Input profiles → Alpamayo2 Super model: model-ready multimodal input
- Conversation templates → Alpamayo2 Super model: prompts
- Text task contracts → Conversation templates: task output contract
- Config utilities → Model configuration: supports
- Model configuration → Alpamayo2 Super model: checkpoint and settings
- Device and distributed runtime → Alpamayo2 Super model: CUDA placement
- Alpamayo2 Super model → Action conditioning projection: backbone conditioning
- Action conditioning projection → Diffusion action expert: projected conditioning
- Diffusion action expert → Flow-matching sampler: denoising model
- Diffusion base → Flow-matching sampler: sampling base
- Action-space contract → Unicycle accel-curvature space: parameterizes
- Unicycle accel-curvature space → Flow-matching sampler: action representation
- Geometry transforms → Flow-matching sampler: trajectory transforms
- Flow-matching sampler → Inference artifacts: sampled trajectories
- Alpamayo2 Super model → Inference artifacts: generated text and metadata
- Smoke inference CLI → PhysicalAI-AV loader: loads samples
- Smoke inference CLI → Alpamayo2 Super model: runs inference
- Smoke inference CLI → Inference artifacts: serializes and evaluates
- Inference artifacts → Visualization API: plot inputs
- Visualization API → Rendering utilities: delegates rendering
- Rendering utilities → Inference artifacts: visualization artifacts
- Inference notebooks → Smoke inference CLI: interactive inference path
- Two-GPU CFG demo → Alpamayo2 Super model: split-GPU CFG execution
- Two-GPU CFG demo → Device and distributed runtime: explicit device control