# apple/ml-starflow architecture

> Architecture diagram of the GitHub repository apple/ml-starflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apple/ml-starflow
- Repository: https://github.com/apple/ml-starflow

## Overview

##### Project type and purpose
- STARFlow/STARFlow-V is an open-source ML framework for text-to-image and text-to-video generation.
- Key use cases: inference (sample.py, scripts/test_*) and training (train.py, scripts/test_train_*).

##### File structure
- Top-level entrypoints: train.py, sample.py, finetune_decoder.py
- Configuration: configs/*.yaml + utils/model_setup.py
- Data handling: dataset.py, scripts/extract_images*.py
- Core model: transformer_flow.py
- VAE & losses: misc/wan_vae2.py, misc/ae_losses.py, misc/lpips.py, misc/discriminator.py
- Utilities: utils/{common.py, inference.py, training.py}
- Scripts for automation: scripts/*.sh, scripts/*.py
- Checkpoint & output storage: ckpts/ (external), assets/ (static), any generated outputs.

##### README insights
- Models: STARFlow (3B parameters) & STARFlow-V (7B parameters)
- Components: T5-XL text encoder, SD-VAE / WAN2.2-VAE, autoregressive flow blocks
- Training: FSDP, mixed precision, block-wise Jacobi sampling
- Sampling: classifier-free guidance, Jacobi iteration for speed.

a. Main components
- CLI Layer: train.py, sample.py, finetune_decoder.py
- Config Loader: configs/*.yaml → utils/model_setup.py
- Data Layer: dataset.py ← scripts/extract_images*
- Model Layer:
· Text Encoder (T5-XL)
· VAE Encoder/Decoder (SD-VAE or WAN2.2-VAE)
· Autoregressive Flow (transformer_flow.py)
- Engine Layer:
· Training engine (utils/training.py) + WandB
· Inference engine (utils/inference.py)
- Storage Layer: ckpts/ for model weights, assets/ & output folders

b. Relationships & interactions
- train.py → model_setup loads config → dataset.py provides batches →
Text Encoder + VAE encode → transformer_flow computes flow →
utils/training orchestrates optimizer, FSDP, loss functions →
checkpoints saved to ckpts/, metrics to WandB.
- sample.py → model_setup loads config & checkpoint →
caption input → Text Encoder + (optional image → VAE encode) →
transformer_flow autoregressive sampling → VAE decode → output media →
utils/inference handles saving and evaluation.

c. Architectural patterns & design principles
- Modular layers: clear separation of CLI, config, data, model, and engines
- Configuration-driven architecture via YAML + model_setup
- Pluggable backbones: easy to swap text encoder or VAE
- Distributed training support (PyTorch FSDP, torchrun)
- Utility modules isolate common code, inference, and training logic.

d. Key technologies & frameworks
- Python & PyTorch (torchrun, FSDP, mixed precision)
- T5-XL (from HuggingFace or internal library)
- SD-VAE and WAN2.2-VAE implementations
- YAML for config, WandB for logging
- Shell scripts for environment & examples.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "CLI Layer"
        Train["train.py"]:::code
        Sample["sample.py"]:::code
        Finetune["finetune_decoder.py"]:::code
    end

    subgraph "Config Loader"
        ConfigYAML["configs/*.yaml"]:::code
        ModelSetup["utils/model_setup.py"]:::code
    end

    subgraph "Data Layer"
        Dataset["dataset.py"]:::code
        ExtractImg["scripts/extract_images.py"]:::code
        ExtractVid["scripts/extract_image_from_video.py"]:::code
    end

    subgraph "Model Layer"
        TextEnc["T5-XL Text Encoder"]:::external
        VAE["WAN2.2-VAE"]:::code
        AE_Loss["AE Losses"]:::code
        LPIPS["LPIPS"]:::code
        Disc["Discriminator"]:::code
        CondUtil["Condition Utils"]:::code
        PosEnc["Positional Encodings"]:::code
        Flow["Autoregressive Flow"]:::code
    end

    subgraph "Engine Layer"
        TrainEng["utils/training.py"]:::code
        InferEng["utils/inference.py"]:::code
        WandB["WandB"]:::service
    end

    subgraph "Storage Layer"
        Assets["assets/"]:::storage
        Checkpoints["ckpts/"]:::storage
    end

    %% CLI to Config
    Train -->|loads| ModelSetup
    Sample -->|loads| ModelSetup
    Finetune -->|loads| ModelSetup
    ModelSetup -->|reads| ConfigYAML

    %% Data ingestion
    Train -->|reads| Dataset
    Sample -->|reads| Dataset
    ExtractImg -->|populates| Dataset
    ExtractVid -->|populates| Dataset

    %% Encoding and Flow
    Dataset --> TextEnc
    Dataset --> VAE
    TextEnc --> Flow
    VAE --> Flow
    CondUtil --> Flow
    PosEnc --> Flow

    %% Losses and training
    Flow --> AE_Loss
    Flow --> LPIPS
    Flow --> Disc
    Flow --> TrainEng
    AE_Loss --> TrainEng
    LPIPS --> TrainEng
    Disc --> TrainEng

    %% Training engine outputs
    TrainEng -->|logs| WandB
    TrainEng -->|saves| Checkpoints

    %% Inference flow
    Sample --> TextEnc
    Sample --> VAE
    Flow --> InferEng
    InferEng -->|writes| Assets

    %% Click Events
    click Train "https://github.com/apple/ml-starflow/blob/main/train.py"
    click Sample "https://github.com/apple/ml-starflow/blob/main/sample.py"
    click Finetune "https://github.com/apple/ml-starflow/blob/main/finetune_decoder.py"
    click ConfigYAML "https://github.com/apple/ml-starflow/blob/main/configs/starflow_3B_t2i_256x256.yaml"
    click ConfigYAML "https://github.com/apple/ml-starflow/blob/main/configs/starflow-v_7B_t2v_caus_480p.yaml"
    click ModelSetup "https://github.com/apple/ml-starflow/blob/main/utils/model_setup.py"
    click Dataset "https://github.com/apple/ml-starflow/blob/main/dataset.py"
    click ExtractImg "https://github.com/apple/ml-starflow/blob/main/scripts/extract_images.py"
    click ExtractVid "https://github.com/apple/ml-starflow/blob/main/scripts/extract_image_from_video.py"
    click Flow "https://github.com/apple/ml-starflow/blob/main/transformer_flow.py"
    click VAE "https://github.com/apple/ml-starflow/blob/main/misc/wan_vae2.py"
    click AE_Loss "https://github.com/apple/ml-starflow/blob/main/misc/ae_losses.py"
    click LPIPS "https://github.com/apple/ml-starflow/blob/main/misc/lpips.py"
    click Disc "https://github.com/apple/ml-starflow/blob/main/misc/discriminator.py"
    click CondUtil "https://github.com/apple/ml-starflow/blob/main/misc/condition_utils.py"
    click PosEnc "https://github.com/apple/ml-starflow/blob/main/misc/pe.py"
    click TrainEng "https://github.com/apple/ml-starflow/blob/main/utils/training.py"
    click InferEng "https://github.com/apple/ml-starflow/blob/main/utils/inference.py"
    click Assets "https://github.com/apple/ml-starflow/tree/main/assets/"
    click Checkpoints "https://github.com/apple/ml-starflow/tree/main/ckpts/"

    %% Styles
    classDef code fill:#D6EFFF,stroke:#0366d6;
    classDef external fill:#D2F8D2,stroke:#28a745;
    classDef storage fill:#FFE8A1,stroke:#d29922;
    classDef service fill:#F0F0F0,stroke:#666,stroke-dasharray:5 5;
```
