How nv-tlabs/lyra works
Project Lyra is a set of generative 3D world-model implementations: Lyra 1 combines diffusion and autoregressive video generation with feed-forward scene reconstruction, while Lyra 2 adds long-horizon explorable worlds and an interactive GUI. The principal user flow is to submit text, image, or video conditioning, run a configured generation or reconstruction model, and produce video or 3D scene output. Training and tokenizer capabilities are included as distinct product branches. Some connections are documented only by the README or source index; unsampled implementation details are not asserted.
Main entry points: world_generation_pipeline.py Diffusion pipeline · model_latent_recon.py Latent reconstruction · server.py GUI server · provider.py Scene data provider
Components 27
World generation
- Diffusion pipeline
Lyra-1/cosmos_predict1/diffusion/inference/world_generation_pipeline.py - Gen3C pipeline
Lyra-1/cosmos_predict1/diffusion/inference/gen3c_pipeline.py - Autoregressive pipeline
Lyra-1/cosmos_predict1/autoregressive/inference/world_generation_pipeline.py - Inference utilities
Lyra-1/cosmos_predict1/diffusion/inference/inference_utils.py - Pipeline base
Lyra-1/cosmos_predict1/utils/base_world_generation_pipeline.py - Prompt upsamplers
Lyra-1/cosmos_predict1/diffusion/prompt_upsampler/inference.py - Lyra-2 inference
Lyra-2/lyra_2/_src/inference/lyra2_ar_inference.py - ZoomGS inference
Lyra-2/lyra_2/_src/inference/lyra2_zoomgs_inference.py
Scene reconstruction
- Latent reconstruction
Lyra-1/src/models/recon/model_latent_recon.py - Scene data provider
Lyra-1/src/models/data/provider.py - Scene model utilities
Lyra-1/src/models/utils/model.py - 3D rendering
Lyra-1/src/models/utils/render.py
Models and training
- Video-world model
Lyra-1/cosmos_predict1/diffusion/model/model_v2w.py - World interpolator
Lyra-1/cosmos_predict1/diffusion/model/model_world_interpolator.py - Autoregressive model
Lyra-1/cosmos_predict1/autoregressive/model.py - Diffusion training
Lyra-1/cosmos_predict1/diffusion/training/train.py - Reconstruction training
Lyra-1/train.py - Lyra-2 model
Lyra-2/lyra_2/_src/models/lyra2_model.py - Lyra-2 network
Lyra-2/lyra_2/_src/networks/wan2pt1_lyra2.py
Safety and tokens
- Content guardrails
Lyra-1/cosmos_predict1/auxiliary/guardrail - Video tokenizer
Lyra-1/cosmos_predict1/tokenizer/inference/video_lib.py
Lyra-2 GUI
- GUI client
Lyra-2/gui/api/client.py - GUI server
Lyra-2/gui/api/server.py - API messages
Lyra-2/gui/api/api_types.py - API serialization
Lyra-2/gui/api/api_serialization.py
Other components
- User
- GUI user
Connections 27
- User → Diffusion pipeline: provides conditioning
README.md - Diffusion pipeline → Inference utilities: uses inference helpers
Lyra-1/cosmos_predict1/diffusion/inference/world_generation_pipeline.py - Diffusion pipeline → Video-world model: loads model
Lyra-1/cosmos_predict1/diffusion/inference/world_generation_pipeline.py - Diffusion pipeline → World interpolator: supports interpolation
Lyra-1/cosmos_predict1/diffusion/inference/world_generation_pipeline.py - Diffusion pipeline → Prompt upsamplers: uses prompt upsamplers
Lyra-1/cosmos_predict1/diffusion/inference/world_generation_pipeline.py - Gen3C pipeline → Diffusion pipeline: extends pipeline
Lyra-1/cosmos_predict1/diffusion/inference/gen3c_pipeline.py - Gen3C pipeline → Inference utilities: uses inference helpers
Lyra-1/cosmos_predict1/diffusion/inference/gen3c_pipeline.py - Autoregressive pipeline → Autoregressive model: runs model
Lyra-1/cosmos_predict1/autoregressive/inference/world_generation_pipeline.py - Autoregressive pipeline → Pipeline base: uses pipeline base
Lyra-1/cosmos_predict1/autoregressive/inference/world_generation_pipeline.py - Pipeline base → Content guardrails: runs safety checks
Lyra-1/cosmos_predict1/utils/base_world_generation_pipeline.py - User → Latent reconstruction: requests reconstruction
README.md - Scene data provider → Latent reconstruction: provides scene data
Lyra-1/src/models/data/provider.py - Latent reconstruction → Scene model utilities: uses model utilities
Lyra-1/src/models/recon/model_latent_recon.py - Latent reconstruction → 3D rendering: uses rendering utilities
Lyra-1/src/models/recon/model_latent_recon.py - Reconstruction training → Latent reconstruction: trains recon model
Lyra-1/train.py - Reconstruction training → Scene data provider: loads data provider
Lyra-1/train.py - Scene data provider → Video tokenizer: uses tokenizer
Lyra-1/src/models/data/provider.py - User → GUI client: submits world request
README.md - GUI client → API serialization: encodes messages
Lyra-2/gui/api/client.py - GUI client → API messages: uses message types
Lyra-2/gui/api/client.py - GUI server → API serialization: decodes requests
Lyra-2/gui/api/server.py - GUI server → API messages: validates message types
Lyra-2/gui/api/server.py - GUI server → Lyra-2 inference: dispatches inference
- Lyra-2 inference → Lyra-2 model: runs model
Lyra-2/lyra_2/_src/inference/lyra2_ar_inference.py - ZoomGS inference → Lyra-2 inference: reuses inference
Lyra-2/lyra_2/_src/inference/lyra2_zoomgs_inference.py - Lyra-2 model → Lyra-2 network: uses network
- Diffusion training → Video-world model: trains diffusion model