lightricks/ltx-2

1. This is a Python monorepo for a foundation video generation system. It is split into three product-level packages: ltx-core for model/runtime primitives, ltx-pipelines for end-user inference workflows, and ltx-trainer for dataset processing and LoRA/fine-tuning.

2. ltx-core is the lowest architectural layer. It contains the reusable model implementation, inference utilities, and performance helpers that pipelines and training both depend on. This package is where checkpoint loading, model composition, conditioning, quantization, and specialized streaming/tiling logic live.

3. The core model layer is divided by modality and model type: a main DiT-style transformer under model/transformer, audio/video VAEs under model/audio_vae and model/video_vae, an upsampler under model/upsampler, and shared model abstractions in model/common and model_protocol.py. This suggests the generation stack is built as latent-space encoding/decoding around a transformer denoiser, with separate components for temporal, spatial, and audio representations.

4. Conditioning is a first-class subsystem in ltx-core. The conditioning package defines typed conditioning inputs such as keyframes, latent conditioning, masks, noise masks, reference audio, and reference video. That boundary is important: pipelines assemble these typed condition objects, while the core model consumes them in a uniform internal format.

5. Inference acceleration and memory management are also centralized in ltx-core. The loader package handles model assembly, attention/kernel selection, LoRA fusion, and single-GPU model building; quantization supports FP8 policies; block_streaming supports streaming model blocks from disk/source/pool abstractions; and tiling/modality_tiling support large-media processing. These are the main runtime boundaries for fitting a large model into GPU memory.

6. ltx-core also includes the text side of the system. The text_encoders/gemma subtree wraps a Gemma-based encoder with tokenizer, feature extraction, prompt files, and embedding connectors. In architecture terms, prompts are converted to text embeddings here before entering the generative model.

7. ltx-pipelines is the orchestration layer for inference modes. Each file in the package corresponds to a specific generation workflow: text-to-video, image-to-video, audio-to-video, distilled fast inference, keyframe interpolation, retake/editing, HDR output, and lip-dub variants. These pipelines combine the core model, text encoder, conditioning objects, denoisers/samplers, and postprocessing into a user-facing execution path.

8. The pipeline layer is where the major data flow is assembled: prompt or media input enters, gets encoded into text/audio/image/video latents or conditions, then a denoising loop runs, often in one or two stages, and finally decoded frames/audio are produced. The README’s emphasis on two-stage pipelines, spatial upscaling, distilled inference, and optional enhancement shows that generation is typically staged rather than a single monolithic forward pass.

9. ltx-trainer is a separate training and fine-tuning subsystem, not just a wrapper around inference. It owns config-driven training setup, dataset preprocessing/captioning scripts, model loading for training, training state/progress tracking, timestep/sigma sampling, and multiple training strategies. This package forms the boundary where raw media data is turned into training examples and then into updated checkpoints or LoRAs.

10. Training is organized around strategy modules rather than one generic loop. training_strategies contains text-to-video, video-to-video, and flexible strategies, which likely determine which inputs are masked/conditioned and what loss path is used. The configs under configs/ and the scripts/ directory indicate that training is expected to run from declarative presets plus preprocessing jobs such as scene splitting, captioning, and latent decoding.

11. The repository depends on external model assets from Hugging Face rather than shipping weights. The runtime expects the main LTX-2 checkpoint, spatial/temporal upscalers, Gemma text encoder assets, and various LoRAs for specialized behaviors such as camera motion, HDR, and lip dubbing. Those assets are not code dependencies; they are runtime inputs that plug into the loader/pipeline boundary.

12. A useful architecture diagram should show three layers: media and prompt inputs at the top; pipeline orchestration in the middle; and ltx-core model/runtime services beneath it. Training should be drawn as a parallel path that consumes prepared datasets and checkpoints to produce LoRA or model artifacts that later feed back into the inference layer.