# sandai-org/magi-1 architecture

> Architecture diagram of the GitHub repository sandai-org/magi-1, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sandai-org/magi-1
- Repository: https://github.com/sandai-org/magi-1

## Overview

##### Project type and purpose
- This is a research-grade, full-stack **machine-learning inference pipeline** for autoregressive video generation (MAGI-1).
- Purpose: load pre-trained Transformer-VAE & T5 models, process user prompts (text, image, or video), perform chunk-wise diffusion denoising to generate video frames, and output a final video.

##### File structure
- Top-level:
- `example/…` – sample configs & run scripts (24B, 4.5B)
- `inference/` – core Python package.
- `requirements.txt`, Docker/Conda setup files, LICENSE, README, figures.
- Inside `inference/`:
- `common/` – configuration, logging, timing, utilities
- `infra/` – checkpointing, distributed context & parallel-state management, parallelism strategies (data/pipe/tile)
- `model/` – implementations for DIT (diffusion transformer), T5 text encoder, VAE modules
- `pipeline/` – entrypoint, prompt processing, video generation and processing logic

##### README for additional insights
- Describes:
- Autoregressive chunk-by-chunk denoising pipeline with up to 4 concurrent chunks
- Core components: Transformer-based VAE, diffusion transformer, distillation algorithm
- Runtime modes: t2v, i2v, v2v controlled via CLI scripts (`run.sh`) and JSON config
- Two recommended environments: Docker image or Conda + pip install

##### Main components
a. **User / CLI**
- Inputs: JSON config, text prompt, optional image/video prefix
- Scripts: `example/*/run.sh`
b. **Config Loader & Common Utils** (`inference/common`)
- Parses `config.json`, sets seeds, logging, timing
c. **Model Manager** (`inference/model`)
- Loads:
- **T5 encoder** for text
- **VAE encoder/decoder** for spatial/temporal compression
- **Diffusion Transformer (DIT)** for chunk denoising
- Fetches weights from checkpoint directory via `infra/checkpoint`
d. **Distributed & Parallelism Infrastructure** (`inference/infra`)
- Initializes distributed backend, process groups
- Applies data parallel, pipeline parallel, or tile parallel as configured
e. **MagiPipeline** (`inference/pipeline/pipeline.py`)
- **Prompt Processor**: tokenizes text, loads prefix media
- **Chunk Scheduler**: orchestrates chunk-wise denoising pipeline (up to 4 concurrent jobs)
- **Denoising Worker**: iterative diffusion sampler invoking DIT model
- **VAE Decoder**: converts latent to frames
- **Video Processor**: assembles frames, post-processing (FFmpeg calls)
f. **Output**
- Final video file, saved at `--output_path`

##### Relationships & Interactions
- CLI → Config Loader → Infra Init → Model Manager → MagiPipeline
- MagiPipeline ↔ Common Utils (logging, timing)
- MagiPipeline dispatches jobs to Denoising Worker which runs inside Parallelism Context
- Workers fetch model weights via Checkpoint module
- Completed frame chunks are passed to VAE Decoder and Video Processor, then concatenated

##### Architectural patterns & design principles
- **Modularization**: clear separation between config, infra, model, pipeline
- **Plugin-style parallelism**: interchangeable data/pipe/tile strategies
- **Chunk-wise streaming**: pipelined generation for low latency and scalability
- **Dependency Injection**: config dataclasses flow into every component
- **Single-responsibility**: each subpackage focuses on one concern

##### Relevant technologies & frameworks
- Python 3.10, PyTorch 2.4.0 + CUDA 12.4
- Docker (optional)
- FFmpeg for video assembly
- MagiAttention (separate GitHub submodule)
- Hugging Face checkpoint storage URLs

##### Elements
- **Labels**: clear component names (e.g., “Diffusion Transformer”, “VAE Decoder”)
- Blue for compute modules (`model/*`, `pipeline/*`)
- Green for data stores (checkpoints, JSON configs, output video)
- Orange for infrastructure (distributed, checkpointing, parallelism)

##### Final advice
- Every ML pipeline stage—from prompt ingestion through denoising to video file write
- Concurrency: chunk-by-chunk scheduling and parallel execution contexts
- Do not overcomplicate: stick to the four main layers (common, infra, model, pipeline), external I/O, and the user entrypoint

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Entrypoint
    subgraph "User / CLI"
        CLI24["24B run.sh"]:::external
        CLI45["4.5B run.sh"]:::external
    end

    %% Config and Common Utilities
    subgraph "Common Utils & Config"
        ConfigJSON["config.json"]:::storage
        ConfigLoader["Config Loader"]:::common
        Utils["Common Utils"]:::common
        Logger["Logger"]:::common
        Timer["Timer"]:::common
        Dataclass["Dataclass"]:::common
    end

    %% Infrastructure Layer
    subgraph "Infrastructure"
        Checkpoint["Checkpointing"]:::infra
        DistUtils["Distributed Utils"]:::infra
        ParallelState["Parallel State"]:::infra
        DataParallel["DataParallel"]:::infra
        PipeParallel["PipelineParallel"]:::infra
        TileParallel["TileParallel"]:::infra
        CheckpointStore["HF / Local Checkpoints"]:::storage
    end

    %% Model Manager
    subgraph "Model Manager"
        T5["T5 Encoder"]:::model
        VAEmod["VAE Model"]:::model
        VAEmodule["VAE Module"]:::model
        DITmod["DIT Model"]:::model
        DITmodule["DIT Module"]:::model
    end

    %% MagiPipeline
    subgraph "MagiPipeline"
        Entry["Entry & CLI Integration"]:::pipeline
        PipelineCore["Core Pipeline (pipeline.py)"]:::pipeline
        Prompt["Prompt Processor"]:::pipeline
        Scheduler["Chunk Scheduler"]:::pipeline
        Worker["Denoising Worker"]:::pipeline
        VAEDec["VAE Decoder"]:::pipeline
        VideoProc["Video Processor"]:::pipeline
    end

    %% Output
    OutputVideo["Output Video"]:::storage

    %% Relationships
    CLI24 --> Entry
    CLI45 --> Entry

    ConfigJSON --> ConfigLoader
    Entry --> ConfigLoader

    ConfigLoader --> Checkpoint
    ConfigLoader --> DistUtils
    ConfigLoader --> ParallelState

    DistUtils --> DataParallel
    DistUtils --> PipeParallel
    DistUtils --> TileParallel

    CheckpointStore --> Checkpoint
    HF["Hugging Face"]:::external --> CheckpointStore

    Checkpoint --> T5
    Checkpoint --> VAEmod
    Checkpoint --> DITmod

    Entry --> PipelineCore
    PipelineCore --> Prompt
    PipelineCore --> Scheduler

    Prompt --> Logger
    Prompt --> Timer

    Scheduler --> Worker
    Worker --> DataParallel
    Worker --> DITmod
    Worker --> DITmodule
    Worker --> VAEmod
    Worker --> VAEmodule

    Worker --> VAEDec
    VAEDec --> VideoProc
    VideoProc --> OutputVideo

    Worker --> Logger
    Worker --> Timer

    %% Click Events
    click CLI24 "https://github.com/sandai-org/magi-1/blob/main/example/24B/run.sh"
    click CLI45 "https://github.com/sandai-org/magi-1/blob/main/example/4.5B/run.sh"
    click ConfigLoader "https://github.com/sandai-org/magi-1/blob/main/inference/common/config.py"
    click Utils "https://github.com/sandai-org/magi-1/blob/main/inference/common/common_utils.py"
    click Logger "https://github.com/sandai-org/magi-1/blob/main/inference/common/logger.py"
    click Timer "https://github.com/sandai-org/magi-1/blob/main/inference/common/timer.py"
    click Dataclass "https://github.com/sandai-org/magi-1/blob/main/inference/common/dataclass.py"
    click Checkpoint "https://github.com/sandai-org/magi-1/blob/main/inference/infra/checkpoint/checkpointing.py"
    click DistUtils "https://github.com/sandai-org/magi-1/blob/main/inference/infra/distributed/dist_utils.py"
    click ParallelState "https://github.com/sandai-org/magi-1/blob/main/inference/infra/distributed/parallel_state.py"
    click DataParallel "https://github.com/sandai-org/magi-1/blob/main/inference/infra/parallelism/context_parallel.py"
    click PipeParallel "https://github.com/sandai-org/magi-1/blob/main/inference/infra/parallelism/pipeline_parallel.py"
    click TileParallel "https://github.com/sandai-org/magi-1/blob/main/inference/infra/parallelism/tile_parallel.py"
    click T5 "https://github.com/sandai-org/magi-1/blob/main/inference/model/t5/t5_model.py"
    click VAEmod "https://github.com/sandai-org/magi-1/blob/main/inference/model/vae/vae_model.py"
    click VAEmodule "https://github.com/sandai-org/magi-1/blob/main/inference/model/vae/vae_module.py"
    click DITmod "https://github.com/sandai-org/magi-1/blob/main/inference/model/dit/dit_model.py"
    click DITmodule "https://github.com/sandai-org/magi-1/blob/main/inference/model/dit/dit_module.py"
    click Entry "https://github.com/sandai-org/magi-1/blob/main/inference/pipeline/entry.py"
    click PipelineCore "https://github.com/sandai-org/magi-1/blob/main/inference/pipeline/pipeline.py"
    click Prompt "https://github.com/sandai-org/magi-1/blob/main/inference/pipeline/prompt_process.py"
    click Worker "https://github.com/sandai-org/magi-1/blob/main/inference/pipeline/video_generate.py"
    click VideoProc "https://github.com/sandai-org/magi-1/blob/main/inference/pipeline/video_process.py"

    %% Styles
    classDef external fill:#f2f2f2,stroke:#333,stroke-width:1px;
    classDef common fill:#cfe2f3,stroke:#215e9f,color:#000;
    classDef infra fill:#f9cb9c,stroke:#b45f06,color:#000;
    classDef model fill:#cfe2f3,stroke:#215e9f,color:#000;
    classDef pipeline fill:#cfe2f3,stroke:#215e9f,color:#000;
    classDef storage fill:#b6d7a8,stroke:#38761d,color:#000;
```
