# nv-tlabs/lyra architecture

> Architecture diagram of the GitHub repository nv-tlabs/lyra, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nv-tlabs/lyra
- Repository: https://github.com/nv-tlabs/lyra

## Overview

Project Lyra is a set of generative 3D world-model implementations: Lyra 1 combines diffusion and autoregressive video generation with feed-forward scene reconstruction, while Lyra 2 adds long-horizon explorable worlds and an interactive GUI. The principal user flow is to submit text, image, or video conditioning, run a configured generation or reconstruction model, and produce video or 3D scene output. Training and tokenizer capabilities are included as distinct product branches. Some connections are documented only by the README or source index; unsampled implementation details are not asserted.

Main entry points: [`Lyra-1/cosmos_predict1/diffusion/inference/world_generation_pipeline.py`](https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/cosmos_predict1/diffusion/inference/world_generation_pipeline.py) (Diffusion pipeline), [`Lyra-1/src/models/recon/model_latent_recon.py`](https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/src/models/recon/model_latent_recon.py) (Latent reconstruction), [`Lyra-2/gui/api/server.py`](https://github.com/nv-tlabs/lyra/blob/main/Lyra-2/gui/api/server.py) (GUI server), [`Lyra-1/src/models/data/provider.py`](https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/src/models/data/provider.py) (Scene data provider)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_generation["World generation"]
  node_lyra1diff["Diffusion pipeline"]
  node_gen3c["Gen3C pipeline<br/>[gen3c_pipeline.py]"]
  node_arpipe["Autoregressive pipeline"]
  node_inferutils["Inference utilities<br/>[inference_utils.py]"]
  node_basepipe["Pipeline base"]
  node_promptup["Prompt upsamplers<br/>[inference.py]"]
  node_lyra2_infer["Lyra-2 inference"]
  node_zoomgs["ZoomGS inference"]
end

subgraph group_reconstruction["Scene reconstruction"]
  node_recon["Latent reconstruction"]
  node_data["Scene data provider<br/>[provider.py]"]
  node_sceneutils["Scene model utilities<br/>[model.py]"]
  node_render["3D rendering<br/>[render.py]"]
end

subgraph group_modeling["Models and training"]
  node_modelv2w["Video-world model<br/>[model_v2w.py]"]
  node_worldinterp["World interpolator"]
  node_ar_model["Autoregressive model<br/>[model.py]"]
  node_train1["Diffusion training<br/>[train.py]"]
  node_scene_train["Reconstruction training<br/>[train.py]"]
  node_lyra2_model["Lyra-2 model<br/>[lyra2_model.py]"]
  node_lyra2_network["Lyra-2 network<br/>[wan2pt1_lyra2.py]"]
end

subgraph group_safety["Safety and tokens"]
  node_guardrails["Content guardrails"]
  node_tokenizer["Video tokenizer<br/>[video_lib.py]"]
end

subgraph group_interactive["Lyra-2 GUI"]
  node_gui_client["GUI client<br/>[client.py]"]
  node_gui_server["GUI server<br/>[server.py]"]
  node_api_types["API messages<br/>[api_types.py]"]
  node_serialization["API serialization"]
end

node_user(("User"))
node_gui_user(("GUI user"))

node_user -.->|"provides conditioning"| node_lyra1diff
node_lyra1diff -->|"uses inference helpers"| node_inferutils
node_lyra1diff -->|"loads model"| node_modelv2w
node_lyra1diff -->|"supports interpolation"| node_worldinterp
node_lyra1diff -->|"uses prompt upsamplers"| node_promptup
node_gen3c -->|"extends pipeline"| node_lyra1diff
node_gen3c -->|"uses inference helpers"| node_inferutils
node_arpipe -->|"runs model"| node_ar_model
node_arpipe -->|"uses pipeline base"| node_basepipe
node_basepipe -->|"runs safety checks"| node_guardrails
node_user -.->|"requests reconstruction"| node_recon
node_data -->|"provides scene data"| node_recon
node_recon -->|"uses model utilities"| node_sceneutils
node_recon -->|"uses rendering utilities"| node_render
node_scene_train -->|"trains recon model"| node_recon
node_scene_train -->|"loads data provider"| node_data
node_data -->|"uses tokenizer"| node_tokenizer
node_user -.->|"submits world request"| node_gui_client
node_gui_client -->|"encodes messages"| node_serialization
node_gui_client -->|"uses message types"| node_api_types
node_gui_server -->|"decodes requests"| node_serialization
node_gui_server -->|"validates message types"| node_api_types
node_gui_server -.->|"dispatches inference"| node_lyra2_infer
node_lyra2_infer -->|"runs model"| node_lyra2_model
node_zoomgs -->|"reuses inference"| node_lyra2_infer
node_lyra2_model -.->|"uses network"| node_lyra2_network
node_train1 -.->|"trains diffusion model"| node_modelv2w

click node_lyra1diff "https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/cosmos_predict1/diffusion/inference/world_generation_pipeline.py"
click node_gen3c "https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/cosmos_predict1/diffusion/inference/gen3c_pipeline.py"
click node_arpipe "https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/cosmos_predict1/autoregressive/inference/world_generation_pipeline.py"
click node_inferutils "https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/cosmos_predict1/diffusion/inference/inference_utils.py"
click node_basepipe "https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/cosmos_predict1/utils/base_world_generation_pipeline.py"
click node_promptup "https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/cosmos_predict1/diffusion/prompt_upsampler/inference.py"
click node_modelv2w "https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/cosmos_predict1/diffusion/model/model_v2w.py"
click node_worldinterp "https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/cosmos_predict1/diffusion/model/model_world_interpolator.py"
click node_ar_model "https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/cosmos_predict1/autoregressive/model.py"
click node_train1 "https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/cosmos_predict1/diffusion/training/train.py"
click node_recon "https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/src/models/recon/model_latent_recon.py"
click node_data "https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/src/models/data/provider.py"
click node_sceneutils "https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/src/models/utils/model.py"
click node_render "https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/src/models/utils/render.py"
click node_scene_train "https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/train.py"
click node_guardrails "https://github.com/nv-tlabs/lyra/tree/main/Lyra-1/cosmos_predict1/auxiliary/guardrail"
click node_tokenizer "https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/cosmos_predict1/tokenizer/inference/video_lib.py"
click node_gui_client "https://github.com/nv-tlabs/lyra/blob/main/Lyra-2/gui/api/client.py"
click node_gui_server "https://github.com/nv-tlabs/lyra/blob/main/Lyra-2/gui/api/server.py"
click node_api_types "https://github.com/nv-tlabs/lyra/blob/main/Lyra-2/gui/api/api_types.py"
click node_serialization "https://github.com/nv-tlabs/lyra/blob/main/Lyra-2/gui/api/api_serialization.py"
click node_lyra2_infer "https://github.com/nv-tlabs/lyra/blob/main/Lyra-2/lyra_2/_src/inference/lyra2_ar_inference.py"
click node_lyra2_model "https://github.com/nv-tlabs/lyra/blob/main/Lyra-2/lyra_2/_src/models/lyra2_model.py"
click node_lyra2_network "https://github.com/nv-tlabs/lyra/blob/main/Lyra-2/lyra_2/_src/networks/wan2pt1_lyra2.py"
click node_zoomgs "https://github.com/nv-tlabs/lyra/blob/main/Lyra-2/lyra_2/_src/inference/lyra2_zoomgs_inference.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_lyra1diff,node_gen3c,node_arpipe,node_inferutils,node_basepipe,node_promptup,node_lyra2_infer,node_zoomgs,node_user,node_gui_user toneBlue
class node_recon,node_data,node_sceneutils,node_render toneAmber
class node_modelv2w,node_worldinterp,node_ar_model,node_train1,node_scene_train,node_lyra2_model,node_lyra2_network toneMint
class node_guardrails,node_tokenizer toneRose
class node_gui_client,node_gui_server,node_api_types,node_serialization toneIndigo
```

## Components

### World generation

- [Diffusion pipeline](https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/cosmos_predict1/diffusion/inference/world_generation_pipeline.py): `Lyra-1/cosmos_predict1/diffusion/inference/world_generation_pipeline.py`
- [Gen3C pipeline](https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/cosmos_predict1/diffusion/inference/gen3c_pipeline.py): `Lyra-1/cosmos_predict1/diffusion/inference/gen3c_pipeline.py`
- [Autoregressive pipeline](https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/cosmos_predict1/autoregressive/inference/world_generation_pipeline.py): `Lyra-1/cosmos_predict1/autoregressive/inference/world_generation_pipeline.py`
- [Inference utilities](https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/cosmos_predict1/diffusion/inference/inference_utils.py): `Lyra-1/cosmos_predict1/diffusion/inference/inference_utils.py`
- [Pipeline base](https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/cosmos_predict1/utils/base_world_generation_pipeline.py): `Lyra-1/cosmos_predict1/utils/base_world_generation_pipeline.py`
- [Prompt upsamplers](https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/cosmos_predict1/diffusion/prompt_upsampler/inference.py): `Lyra-1/cosmos_predict1/diffusion/prompt_upsampler/inference.py`
- [Lyra-2 inference](https://github.com/nv-tlabs/lyra/blob/main/Lyra-2/lyra_2/_src/inference/lyra2_ar_inference.py): `Lyra-2/lyra_2/_src/inference/lyra2_ar_inference.py`
- [ZoomGS inference](https://github.com/nv-tlabs/lyra/blob/main/Lyra-2/lyra_2/_src/inference/lyra2_zoomgs_inference.py): `Lyra-2/lyra_2/_src/inference/lyra2_zoomgs_inference.py`

### Scene reconstruction

- [Latent reconstruction](https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/src/models/recon/model_latent_recon.py): `Lyra-1/src/models/recon/model_latent_recon.py`
- [Scene data provider](https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/src/models/data/provider.py): `Lyra-1/src/models/data/provider.py`
- [Scene model utilities](https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/src/models/utils/model.py): `Lyra-1/src/models/utils/model.py`
- [3D rendering](https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/src/models/utils/render.py): `Lyra-1/src/models/utils/render.py`

### Models and training

- [Video-world model](https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/cosmos_predict1/diffusion/model/model_v2w.py): `Lyra-1/cosmos_predict1/diffusion/model/model_v2w.py`
- [World interpolator](https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/cosmos_predict1/diffusion/model/model_world_interpolator.py): `Lyra-1/cosmos_predict1/diffusion/model/model_world_interpolator.py`
- [Autoregressive model](https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/cosmos_predict1/autoregressive/model.py): `Lyra-1/cosmos_predict1/autoregressive/model.py`
- [Diffusion training](https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/cosmos_predict1/diffusion/training/train.py): `Lyra-1/cosmos_predict1/diffusion/training/train.py`
- [Reconstruction training](https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/train.py): `Lyra-1/train.py`
- [Lyra-2 model](https://github.com/nv-tlabs/lyra/blob/main/Lyra-2/lyra_2/_src/models/lyra2_model.py): `Lyra-2/lyra_2/_src/models/lyra2_model.py`
- [Lyra-2 network](https://github.com/nv-tlabs/lyra/blob/main/Lyra-2/lyra_2/_src/networks/wan2pt1_lyra2.py): `Lyra-2/lyra_2/_src/networks/wan2pt1_lyra2.py`

### Safety and tokens

- [Content guardrails](https://github.com/nv-tlabs/lyra/tree/main/Lyra-1/cosmos_predict1/auxiliary/guardrail): `Lyra-1/cosmos_predict1/auxiliary/guardrail`
- [Video tokenizer](https://github.com/nv-tlabs/lyra/blob/main/Lyra-1/cosmos_predict1/tokenizer/inference/video_lib.py): `Lyra-1/cosmos_predict1/tokenizer/inference/video_lib.py`

### Lyra-2 GUI

- [GUI client](https://github.com/nv-tlabs/lyra/blob/main/Lyra-2/gui/api/client.py): `Lyra-2/gui/api/client.py`
- [GUI server](https://github.com/nv-tlabs/lyra/blob/main/Lyra-2/gui/api/server.py): `Lyra-2/gui/api/server.py`
- [API messages](https://github.com/nv-tlabs/lyra/blob/main/Lyra-2/gui/api/api_types.py): `Lyra-2/gui/api/api_types.py`
- [API serialization](https://github.com/nv-tlabs/lyra/blob/main/Lyra-2/gui/api/api_serialization.py): `Lyra-2/gui/api/api_serialization.py`

### Other components

- **User**
- **GUI user**

## Connections

- User → Diffusion pipeline: provides conditioning (evidence: `README.md`)
- Diffusion pipeline → Inference utilities: uses inference helpers (evidence: `Lyra-1/cosmos_predict1/diffusion/inference/world_generation_pipeline.py`)
- Diffusion pipeline → Video-world model: loads model (evidence: `Lyra-1/cosmos_predict1/diffusion/inference/world_generation_pipeline.py`)
- Diffusion pipeline → World interpolator: supports interpolation (evidence: `Lyra-1/cosmos_predict1/diffusion/inference/world_generation_pipeline.py`)
- Diffusion pipeline → Prompt upsamplers: uses prompt upsamplers (evidence: `Lyra-1/cosmos_predict1/diffusion/inference/world_generation_pipeline.py`)
- Gen3C pipeline → Diffusion pipeline: extends pipeline (evidence: `Lyra-1/cosmos_predict1/diffusion/inference/gen3c_pipeline.py`)
- Gen3C pipeline → Inference utilities: uses inference helpers (evidence: `Lyra-1/cosmos_predict1/diffusion/inference/gen3c_pipeline.py`)
- Autoregressive pipeline → Autoregressive model: runs model (evidence: `Lyra-1/cosmos_predict1/autoregressive/inference/world_generation_pipeline.py`)
- Autoregressive pipeline → Pipeline base: uses pipeline base (evidence: `Lyra-1/cosmos_predict1/autoregressive/inference/world_generation_pipeline.py`)
- Pipeline base → Content guardrails: runs safety checks (evidence: `Lyra-1/cosmos_predict1/utils/base_world_generation_pipeline.py`)
- User → Latent reconstruction: requests reconstruction (evidence: `README.md`)
- Scene data provider → Latent reconstruction: provides scene data (evidence: `Lyra-1/src/models/data/provider.py`)
- Latent reconstruction → Scene model utilities: uses model utilities (evidence: `Lyra-1/src/models/recon/model_latent_recon.py`)
- Latent reconstruction → 3D rendering: uses rendering utilities (evidence: `Lyra-1/src/models/recon/model_latent_recon.py`)
- Reconstruction training → Latent reconstruction: trains recon model (evidence: `Lyra-1/train.py`)
- Reconstruction training → Scene data provider: loads data provider (evidence: `Lyra-1/train.py`)
- Scene data provider → Video tokenizer: uses tokenizer (evidence: `Lyra-1/src/models/data/provider.py`)
- User → GUI client: submits world request (evidence: `README.md`)
- GUI client → API serialization: encodes messages (evidence: `Lyra-2/gui/api/client.py`)
- GUI client → API messages: uses message types (evidence: `Lyra-2/gui/api/client.py`)
- GUI server → API serialization: decodes requests (evidence: `Lyra-2/gui/api/server.py`)
- GUI server → API messages: validates message types (evidence: `Lyra-2/gui/api/server.py`)
- GUI server → Lyra-2 inference: dispatches inference
- Lyra-2 inference → Lyra-2 model: runs model (evidence: `Lyra-2/lyra_2/_src/inference/lyra2_ar_inference.py`)
- ZoomGS inference → Lyra-2 inference: reuses inference (evidence: `Lyra-2/lyra_2/_src/inference/lyra2_zoomgs_inference.py`)
- Lyra-2 model → Lyra-2 network: uses network
- Diffusion training → Video-world model: trains diffusion model
