# nvidia/cosmos architecture

> Architecture diagram of the GitHub repository nvidia/cosmos, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvidia/cosmos
- Repository: https://github.com/nvidia/cosmos

## Overview

Cosmos is a model platform for physical-AI generation and reasoning; the supplied source excerpts chiefly expose its VLMEvalKit evaluation workflow. An evaluator selects datasets and models, runs inference, saves predictions, then scores them. The README also documents generator and reasoner use, while the tree includes generator evaluation suites. Core model-generation implementation is not shown, so those capabilities are represented only at the documented level; links without direct evidence are omitted.

Main entry points: [`evaluation/cosmos3/reasoner/vlmevalkit/run.py`](https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/run.py) (VLMEvalKit runner), [`evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/__init__.py`](https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/__init__.py) (Dataset registry), [`evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/inference.py`](https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/inference.py) (Inference runners), [`evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/config.py`](https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/config.py) (Model configuration)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_eval["Evaluation orchestration"]
  node_entry["VLMEvalKit runner<br/>[run.py]"]
  node_config["Model configuration<br/>[config.py]"]
  node_infer["Inference runners<br/>[inference.py]"]
  node_video_infer["Video inference<br/>[inference_video.py]"]
  node_api_pipeline["API task pipeline<br/>[inference_api.py]"]
  node_file_utils["Result file utilities<br/>[file.py]"]
end

subgraph group_bench["Benchmarks and scoring"]
  node_dataset_registry["Dataset registry<br/>[__init__.py]"]
  node_image_bench["Image benchmarks<br/>[image_vqa.py]"]
  node_cgbench["CG benchmark<br/>[cgbench.py]"]
  node_chartmimic["Chart benchmark<br/>[chartmimic.py]"]
  node_creation["Creation benchmark<br/>[creation.py]"]
  node_answer_extract["Answer extraction<br/>[llm_extract.py]"]
end

subgraph group_models["Model interfaces"]
  node_cosmos_api["Cosmos Reason API<br/>[cosmos_reason.py]"]
  node_local_api["LMDeploy API<br/>[lmdeploy.py]"]
  node_spatial_model["Spatial MLLM<br/>[spatial_mllm.py]"]
  node_lvs_api["LVS service adapter<br/>[lvs_service.py]"]
end

subgraph group_gen_eval["Generator evaluation"]
  node_paibench["PAIBench scoring"]
  node_unigenbench["UniGenBench scoring<br/>[ugb_scorer.py]"]
  node_unigen_utils["UniGen utilities<br/>[utils.py]"]
  node_score_parser["Cosmos score parsing<br/>[parse_score.py]"]
end

node_user(("Evaluator"))
node_generator["Cosmos generator"]

node_user -->|"starts evaluation"| node_entry
node_entry -->|"loads configuration"| node_config
node_entry -->|"selects datasets"| node_dataset_registry
node_entry -->|"dispatches inference"| node_infer
node_entry -->|"dispatches video"| node_video_infer
node_config -->|"configures API"| node_cosmos_api
node_infer -->|"builds prompts"| node_dataset_registry
node_infer -->|"loads prior results"| node_file_utils
node_video_infer -->|"resumes results"| node_file_utils
node_api_pipeline -->|"gets benchmark data"| node_dataset_registry
node_dataset_registry -->|"registers benchmark"| node_image_bench
node_image_bench -->|"uses extraction"| node_answer_extract
node_generator -.->|"evaluated by"| node_paibench
node_generator -.->|"evaluated by"| node_unigenbench
node_unigenbench -->|"uses utilities"| node_unigen_utils
node_user -->|"runs evaluation"| node_paibench
node_user -.->|"parses scores"| node_score_parser

click node_entry "https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/run.py"
click node_config "https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/config.py"
click node_infer "https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/inference.py"
click node_video_infer "https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/inference_video.py"
click node_api_pipeline "https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/inference_api.py"
click node_dataset_registry "https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/__init__.py"
click node_image_bench "https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/image_vqa.py"
click node_cgbench "https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/cgbench.py"
click node_chartmimic "https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/chartmimic.py"
click node_creation "https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/creation.py"
click node_answer_extract "https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/utils/spatial_bench/llm_extract.py"
click node_file_utils "https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/smp/file.py"
click node_cosmos_api "https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/api/cosmos_reason.py"
click node_local_api "https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/api/lmdeploy.py"
click node_spatial_model "https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/vlm/spatial_mllm.py"
click node_lvs_api "https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/api/lvs_service.py"
click node_paibench "https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/generator/paibench_g/run_motion_smoothness_sharded.py"
click node_unigenbench "https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/generator/unigenbench/ugb_scorer.py"
click node_unigen_utils "https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/generator/unigenbench/utils.py"
click node_score_parser "https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/cosmos_eval/parse_score.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_entry,node_config,node_infer,node_video_infer,node_api_pipeline,node_file_utils toneBlue
class node_dataset_registry,node_image_bench,node_cgbench,node_chartmimic,node_creation,node_answer_extract toneAmber
class node_cosmos_api,node_local_api,node_spatial_model,node_lvs_api toneMint
class node_paibench,node_unigenbench,node_unigen_utils,node_score_parser toneRose
class node_user,node_generator toneIndigo
```

## Components

### Evaluation orchestration

- [VLMEvalKit runner](https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/run.py): `evaluation/cosmos3/reasoner/vlmevalkit/run.py`
- [Model configuration](https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/config.py): `evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/config.py`
- [Inference runners](https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/inference.py): `evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/inference.py`
- [Video inference](https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/inference_video.py): `evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/inference_video.py`
- [API task pipeline](https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/inference_api.py): `evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/inference_api.py`
- [Result file utilities](https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/smp/file.py): `evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/smp/file.py`

### Benchmarks and scoring

- [Dataset registry](https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/__init__.py): `evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/__init__.py`
- [Image benchmarks](https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/image_vqa.py): `evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/image_vqa.py`
- [CG benchmark](https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/cgbench.py): `evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/cgbench.py`
- [Chart benchmark](https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/chartmimic.py): `evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/chartmimic.py`
- [Creation benchmark](https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/creation.py): `evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/creation.py`
- [Answer extraction](https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/utils/spatial_bench/llm_extract.py): `evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/utils/spatial_bench/llm_extract.py`

### Model interfaces

- [Cosmos Reason API](https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/api/cosmos_reason.py): `evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/api/cosmos_reason.py`
- [LMDeploy API](https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/api/lmdeploy.py): `evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/api/lmdeploy.py`
- [Spatial MLLM](https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/vlm/spatial_mllm.py): `evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/vlm/spatial_mllm.py`
- [LVS service adapter](https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/api/lvs_service.py): `evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/api/lvs_service.py`

### Generator evaluation

- [PAIBench scoring](https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/generator/paibench_g/run_motion_smoothness_sharded.py): `evaluation/cosmos3/generator/paibench_g/run_motion_smoothness_sharded.py`
- [UniGenBench scoring](https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/generator/unigenbench/ugb_scorer.py): `evaluation/cosmos3/generator/unigenbench/ugb_scorer.py`
- [UniGen utilities](https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/generator/unigenbench/utils.py): `evaluation/cosmos3/generator/unigenbench/utils.py`
- [Cosmos score parsing](https://github.com/nvidia/cosmos/blob/main/evaluation/cosmos3/reasoner/vlmevalkit/cosmos_eval/parse_score.py): `evaluation/cosmos3/reasoner/vlmevalkit/cosmos_eval/parse_score.py`

### Other components

- **Evaluator**
- **Cosmos generator**

## Connections

- Evaluator → VLMEvalKit runner: starts evaluation (evidence: `evaluation/cosmos3/reasoner/vlmevalkit/run.py`)
- VLMEvalKit runner → Model configuration: loads configuration (evidence: `evaluation/cosmos3/reasoner/vlmevalkit/run.py`)
- VLMEvalKit runner → Dataset registry: selects datasets (evidence: `evaluation/cosmos3/reasoner/vlmevalkit/run.py`)
- VLMEvalKit runner → Inference runners: dispatches inference (evidence: `evaluation/cosmos3/reasoner/vlmevalkit/run.py`)
- VLMEvalKit runner → Video inference: dispatches video (evidence: `evaluation/cosmos3/reasoner/vlmevalkit/run.py`)
- Model configuration → Cosmos Reason API: configures API (evidence: `evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/config.py`)
- Inference runners → Dataset registry: builds prompts (evidence: `evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/inference.py`)
- Inference runners → Result file utilities: loads prior results (evidence: `evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/inference.py`)
- Video inference → Result file utilities: resumes results (evidence: `evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/inference_video.py`)
- API task pipeline → Dataset registry: gets benchmark data (evidence: `evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/inference_api.py`)
- Dataset registry → Image benchmarks: registers benchmark (evidence: `evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/__init__.py`)
- Image benchmarks → Answer extraction: uses extraction (evidence: `evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/image_vqa.py`)
- Cosmos generator → PAIBench scoring: evaluated by (evidence: `README.md`)
- Cosmos generator → UniGenBench scoring: evaluated by
- UniGenBench scoring → UniGen utilities: uses utilities (evidence: `evaluation/cosmos3/generator/unigenbench/ugb_scorer.py`)
- Evaluator → PAIBench scoring: runs evaluation (evidence: `README.md`)
- Evaluator → Cosmos score parsing: parses scores
