# lightricks/ltx-2 architecture

> Architecture diagram of the GitHub repository lightricks/ltx-2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lightricks/ltx-2
- Repository: https://github.com/lightricks/ltx-2

## Overview

1. This is a Python monorepo for a foundation video generation system. It is split into three product-level packages: `ltx-core` for model/runtime primitives, `ltx-pipelines` for end-user inference workflows, and `ltx-trainer` for dataset processing and LoRA/fine-tuning.

2. `ltx-core` is the lowest architectural layer. It contains the reusable model implementation, inference utilities, and performance helpers that pipelines and training both depend on. This package is where checkpoint loading, model composition, conditioning, quantization, and specialized streaming/tiling logic live.

3. The core model layer is divided by modality and model type: a main DiT-style transformer under `model/transformer`, audio/video VAEs under `model/audio_vae` and `model/video_vae`, an upsampler under `model/upsampler`, and shared model abstractions in `model/common` and `model_protocol.py`. This suggests the generation stack is built as latent-space encoding/decoding around a transformer denoiser, with separate components for temporal, spatial, and audio representations.

4. Conditioning is a first-class subsystem in `ltx-core`. The `conditioning` package defines typed conditioning inputs such as keyframes, latent conditioning, masks, noise masks, reference audio, and reference video. That boundary is important: pipelines assemble these typed condition objects, while the core model consumes them in a uniform internal format.

5. Inference acceleration and memory management are also centralized in `ltx-core`. The `loader` package handles model assembly, attention/kernel selection, LoRA fusion, and single-GPU model building; `quantization` supports FP8 policies; `block_streaming` supports streaming model blocks from disk/source/pool abstractions; and `tiling`/`modality_tiling` support large-media processing. These are the main runtime boundaries for fitting a large model into GPU memory.

6. `ltx-core` also includes the text side of the system. The `text_encoders/gemma` subtree wraps a Gemma-based encoder with tokenizer, feature extraction, prompt files, and embedding connectors. In architecture terms, prompts are converted to text embeddings here before entering the generative model.

7. `ltx-pipelines` is the orchestration layer for inference modes. Each file in the package corresponds to a specific generation workflow: text-to-video, image-to-video, audio-to-video, distilled fast inference, keyframe interpolation, retake/editing, HDR output, and lip-dub variants. These pipelines combine the core model, text encoder, conditioning objects, denoisers/samplers, and postprocessing into a user-facing execution path.

8. The pipeline layer is where the major data flow is assembled: prompt or media input enters, gets encoded into text/audio/image/video latents or conditions, then a denoising loop runs, often in one or two stages, and finally decoded frames/audio are produced. The README’s emphasis on two-stage pipelines, spatial upscaling, distilled inference, and optional enhancement shows that generation is typically staged rather than a single monolithic forward pass.

9. `ltx-trainer` is a separate training and fine-tuning subsystem, not just a wrapper around inference. It owns config-driven training setup, dataset preprocessing/captioning scripts, model loading for training, training state/progress tracking, timestep/sigma sampling, and multiple training strategies. This package forms the boundary where raw media data is turned into training examples and then into updated checkpoints or LoRAs.

10. Training is organized around strategy modules rather than one generic loop. `training_strategies` contains text-to-video, video-to-video, and flexible strategies, which likely determine which inputs are masked/conditioned and what loss path is used. The configs under `configs/` and the `scripts/` directory indicate that training is expected to run from declarative presets plus preprocessing jobs such as scene splitting, captioning, and latent decoding.

11. The repository depends on external model assets from Hugging Face rather than shipping weights. The runtime expects the main LTX-2 checkpoint, spatial/temporal upscalers, Gemma text encoder assets, and various LoRAs for specialized behaviors such as camera motion, HDR, and lip dubbing. Those assets are not code dependencies; they are runtime inputs that plug into the loader/pipeline boundary.

12. A useful architecture diagram should show three layers: media and prompt inputs at the top; pipeline orchestration in the middle; and `ltx-core` model/runtime services beneath it. Training should be drawn as a parallel path that consumes prepared datasets and checkpoints to produce LoRA or model artifacts that later feed back into the inference layer.

Main entry points: [`packages/ltx-core/src/ltx_core/loader`](https://github.com/lightricks/ltx-2/tree/main/packages/ltx-core/src/ltx_core/loader) (Model loader), [`packages/ltx-core/src/ltx_core/model/transformer`](https://github.com/lightricks/ltx-2/tree/main/packages/ltx-core/src/ltx_core/model/transformer) (Transformer), [`packages/ltx-trainer/src/ltx_trainer/trainer.py`](https://github.com/lightricks/ltx-2/blob/main/packages/ltx-trainer/src/ltx_trainer/trainer.py) (Trainer), [`packages/ltx-pipelines/src/ltx_pipelines/utils`](https://github.com/lightricks/ltx-2/tree/main/packages/ltx-pipelines/src/ltx_pipelines/utils) (Pipeline utils)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_repo["ltx-2"]
  node_inputs["Media + prompts<br/>external inputs"]
  node_artifacts[("Checkpoints + LoRAs<br/>model artifacts")]
end

subgraph group_core["ltx-core"]
  node_core_loader["Model loader<br/>runtime assembly"]
  node_core_conditioning["Conditioning<br/>typed conditions"]
  node_core_text["Gemma text<br/>text encoder"]
  node_core_transformer["Transformer<br/>DiT model"]
  node_core_video_vae["Video VAE<br/>video latent codec"]
  node_core_audio_vae["Audio VAE<br/>audio latent codec"]
  node_core_upsampler["Upsampler<br/>spatial scaler"]
  node_core_runtime["Runtime helpers<br/>acceleration utils"]
end

subgraph group_pipelines["ltx-pipelines"]
  node_pipe_t2v["T2V pipeline<br/>inference workflow"]
  node_pipe_i2v["I2V pipeline<br/>inference workflow"]
  node_pipe_a2vid["A2VID pipeline<br/>inference workflow<br/>[a2vid_two_stage.py]"]
  node_pipe_edit["Edit flows<br/>inference workflow"]
  node_pipe_utils["Pipeline utils<br/>orchestration helpers"]
end

subgraph group_trainer["ltx-trainer"]
  node_train_data["Dataset prep<br/>data pipeline"]
  node_train_loader["Train loader<br/>training setup<br/>[model_loader.py]"]
  node_train_loop["Trainer<br/>training loop<br/>[trainer.py]"]
  node_train_strategies["Strategies<br/>loss/path variants"]
  node_train_sampling["Sampling + state<br/>training control"]
end

node_inputs -->|"prompt text"| node_core_text
node_inputs -->|"media conditions"| node_core_conditioning
node_core_text -->|"embeddings"| node_pipe_utils
node_core_conditioning -->|"typed conditions"| node_pipe_utils
node_pipe_utils -->|"assemble model"| node_core_loader
node_core_loader -->|"build denoiser"| node_core_transformer
node_core_loader -->|"load codec"| node_core_video_vae
node_core_loader -->|"load codec"| node_core_audio_vae
node_core_loader -->|"load scaler"| node_core_upsampler
node_pipe_t2v -->|"denoise"| node_core_transformer
node_pipe_i2v -->|"denoise"| node_core_transformer
node_pipe_a2vid -->|"denoise"| node_core_transformer
node_pipe_edit -->|"denoise"| node_core_transformer
node_pipe_t2v -->|"decode frames"| node_core_video_vae
node_pipe_i2v -->|"decode frames"| node_core_video_vae
node_pipe_a2vid -->|"decode audio"| node_core_audio_vae
node_pipe_edit -->|"enhance output"| node_core_upsampler
node_core_runtime -.->|"stream/quantize"| node_core_loader
node_core_runtime -.->|"tile/accelerate"| node_core_transformer
node_train_data -->|"prepared data"| node_train_loader
node_train_loader -->|"training model"| node_train_loop
node_train_loop -->|"select strategy"| node_train_strategies
node_train_loop -->|"schedule + state"| node_train_sampling
node_train_loop -->|"emit checkpoints"| node_artifacts
node_artifacts -.->|"reload LoRA/checkpoint"| node_core_loader
node_artifacts -.->|"runtime weights"| node_pipe_utils

click node_core_loader "https://github.com/lightricks/ltx-2/tree/main/packages/ltx-core/src/ltx_core/loader"
click node_core_conditioning "https://github.com/lightricks/ltx-2/tree/main/packages/ltx-core/src/ltx_core/conditioning"
click node_core_text "https://github.com/lightricks/ltx-2/tree/main/packages/ltx-core/src/ltx_core/text_encoders/gemma"
click node_core_transformer "https://github.com/lightricks/ltx-2/tree/main/packages/ltx-core/src/ltx_core/model/transformer"
click node_core_video_vae "https://github.com/lightricks/ltx-2/tree/main/packages/ltx-core/src/ltx_core/model/video_vae"
click node_core_audio_vae "https://github.com/lightricks/ltx-2/tree/main/packages/ltx-core/src/ltx_core/model/audio_vae"
click node_core_upsampler "https://github.com/lightricks/ltx-2/tree/main/packages/ltx-core/src/ltx_core/model/upsampler"
click node_core_runtime "https://github.com/lightricks/ltx-2/tree/main/packages/ltx-core/src/ltx_core"
click node_pipe_t2v "https://github.com/lightricks/ltx-2/blob/main/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_one_stage.py"
click node_pipe_i2v "https://github.com/lightricks/ltx-2/blob/main/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages.py"
click node_pipe_a2vid "https://github.com/lightricks/ltx-2/blob/main/packages/ltx-pipelines/src/ltx_pipelines/a2vid_two_stage.py"
click node_pipe_edit "https://github.com/lightricks/ltx-2/tree/main/packages/ltx-pipelines/src/ltx_pipelines"
click node_pipe_utils "https://github.com/lightricks/ltx-2/tree/main/packages/ltx-pipelines/src/ltx_pipelines/utils"
click node_train_data "https://github.com/lightricks/ltx-2/tree/main/packages/ltx-trainer/scripts"
click node_train_loader "https://github.com/lightricks/ltx-2/blob/main/packages/ltx-trainer/src/ltx_trainer/model_loader.py"
click node_train_loop "https://github.com/lightricks/ltx-2/blob/main/packages/ltx-trainer/src/ltx_trainer/trainer.py"
click node_train_strategies "https://github.com/lightricks/ltx-2/tree/main/packages/ltx-trainer/src/ltx_trainer/training_strategies"
click node_train_sampling "https://github.com/lightricks/ltx-2/tree/main/packages/ltx-trainer/src/ltx_trainer"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_inputs,node_artifacts toneBlue
class node_core_loader,node_core_conditioning,node_core_text,node_core_transformer,node_core_video_vae,node_core_audio_vae,node_core_upsampler,node_core_runtime toneAmber
class node_pipe_t2v,node_pipe_i2v,node_pipe_a2vid,node_pipe_edit,node_pipe_utils toneMint
class node_train_data,node_train_loader,node_train_loop,node_train_strategies,node_train_sampling toneRose
```

## Components

### ltx-2

Monorepo for generation runtime, inference pipelines, and training.

- **Media + prompts**: external inputs · Text prompts and media assets that enter pipelines or training prep.
- **Checkpoints + LoRAs**: model artifacts · Produced training artifacts that feed back into inference loading.

### ltx-core

Shared model and runtime primitives.

- [Model loader](https://github.com/lightricks/ltx-2/tree/main/packages/ltx-core/src/ltx_core/loader): runtime assembly · Builds models, selects kernels, fuses LoRAs, and manages attention/runtime ops. · `packages/ltx-core/src/ltx_core/loader`
- [Conditioning](https://github.com/lightricks/ltx-2/tree/main/packages/ltx-core/src/ltx_core/conditioning): typed conditions · Keyframes, latent conditions, masks, noise masks, reference audio/video, and related wrappers. · `packages/ltx-core/src/ltx_core/conditioning`
- [Gemma text](https://github.com/lightricks/ltx-2/tree/main/packages/ltx-core/src/ltx_core/text_encoders/gemma): text encoder · Tokenizer, feature extraction, prompt handling, and embedding connectors for text conditioning. · `packages/ltx-core/src/ltx_core/text_encoders/gemma`
- [Transformer](https://github.com/lightricks/ltx-2/tree/main/packages/ltx-core/src/ltx_core/model/transformer): DiT model · Main denoising transformer and shared transformer utilities. · `packages/ltx-core/src/ltx_core/model/transformer`
- [Video VAE](https://github.com/lightricks/ltx-2/tree/main/packages/ltx-core/src/ltx_core/model/video_vae): video latent codec · Video encoding/decoding, tiling, and memory-efficient decode paths. · `packages/ltx-core/src/ltx_core/model/video_vae`
- [Audio VAE](https://github.com/lightricks/ltx-2/tree/main/packages/ltx-core/src/ltx_core/model/audio_vae): audio latent codec · Audio encoding/decoding and vocoder-related components. · `packages/ltx-core/src/ltx_core/model/audio_vae`
- [Upsampler](https://github.com/lightricks/ltx-2/tree/main/packages/ltx-core/src/ltx_core/model/upsampler): spatial scaler · Spatial super-resolution and resampling components. · `packages/ltx-core/src/ltx_core/model/upsampler`
- [Runtime helpers](https://github.com/lightricks/ltx-2/tree/main/packages/ltx-core/src/ltx_core): acceleration utils · Block streaming, tiling, quantization, schedulers, and guidance helpers used to fit models into memory and run efficiently. · `packages/ltx-core/src/ltx_core`

### ltx-pipelines

Inference workflow orchestration.

- [T2V pipeline](https://github.com/lightricks/ltx-2/blob/main/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_one_stage.py): inference workflow · Text-to-video generation flow. · `packages/ltx-pipelines/src/ltx_pipelines/ti2vid_one_stage.py`
- [I2V pipeline](https://github.com/lightricks/ltx-2/blob/main/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages.py): inference workflow · Image-conditioned video generation flow. · `packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages.py`
- [A2VID pipeline](https://github.com/lightricks/ltx-2/blob/main/packages/ltx-pipelines/src/ltx_pipelines/a2vid_two_stage.py): inference workflow · Audio-conditioned video generation flow, including two-stage variants. · `packages/ltx-pipelines/src/ltx_pipelines/a2vid_two_stage.py`
- [Edit flows](https://github.com/lightricks/ltx-2/tree/main/packages/ltx-pipelines/src/ltx_pipelines): inference workflow · Retake, keyframe interpolation, HDR, lip-dub, and related specialized generation paths. · `packages/ltx-pipelines/src/ltx_pipelines`
- [Pipeline utils](https://github.com/lightricks/ltx-2/tree/main/packages/ltx-pipelines/src/ltx_pipelines/utils): orchestration helpers · Sampling, denoising, media I/O, GPU model wrappers, and quantization selection for pipelines. · `packages/ltx-pipelines/src/ltx_pipelines/utils`

### ltx-trainer

Dataset processing, training, and fine-tuning.

- [Dataset prep](https://github.com/lightricks/ltx-2/tree/main/packages/ltx-trainer/scripts): data pipeline · Captioning, scene splitting, preprocessing, latent decoding, and reference computation. · `packages/ltx-trainer/scripts`
- [Train loader](https://github.com/lightricks/ltx-2/blob/main/packages/ltx-trainer/src/ltx_trainer/model_loader.py): training setup · Loads base models, HF assets, and training-time model variants. · `packages/ltx-trainer/src/ltx_trainer/model_loader.py`
- [Trainer](https://github.com/lightricks/ltx-2/blob/main/packages/ltx-trainer/src/ltx_trainer/trainer.py): training loop · Config-driven training execution and progress/state management. · `packages/ltx-trainer/src/ltx_trainer/trainer.py`
- [Strategies](https://github.com/lightricks/ltx-2/tree/main/packages/ltx-trainer/src/ltx_trainer/training_strategies): loss/path variants · Text-to-video, video-to-video, and flexible strategies that define training behavior. · `packages/ltx-trainer/src/ltx_trainer/training_strategies`
- [Sampling + state](https://github.com/lightricks/ltx-2/tree/main/packages/ltx-trainer/src/ltx_trainer): training control · Timestep/sigma sampling, state tracking, and validation/progress helpers. · `packages/ltx-trainer/src/ltx_trainer`

## Connections

- Media + prompts → Gemma text: prompt text
- Media + prompts → Conditioning: media conditions
- Gemma text → Pipeline utils: embeddings
- Conditioning → Pipeline utils: typed conditions
- Pipeline utils → Model loader: assemble model
- Model loader → Transformer: build denoiser
- Model loader → Video VAE: load codec
- Model loader → Audio VAE: load codec
- Model loader → Upsampler: load scaler
- T2V pipeline → Transformer: denoise
- I2V pipeline → Transformer: denoise
- A2VID pipeline → Transformer: denoise
- Edit flows → Transformer: denoise
- T2V pipeline → Video VAE: decode frames
- I2V pipeline → Video VAE: decode frames
- A2VID pipeline → Audio VAE: decode audio
- Edit flows → Upsampler: enhance output
- Runtime helpers → Model loader: stream/quantize
- Runtime helpers → Transformer: tile/accelerate
- Dataset prep → Train loader: prepared data
- Train loader → Trainer: training model
- Trainer → Strategies: select strategy
- Trainer → Sampling + state: schedule + state
- Trainer → Checkpoints + LoRAs: emit checkpoints
- Checkpoints + LoRAs → Model loader: reload LoRA/checkpoint
- Checkpoints + LoRAs → Pipeline utils: runtime weights