How apple/ml-starflow works
Project type and purpose
• STARFlow/STARFlow-V is an open-source ML framework for text-to-image and text-to-video generation.
• Key use cases: inference (sample.py, scripts/test_*) and training (train.py, scripts/test_train_*).
File structure
• Top-level entrypoints: train.py, sample.py, finetune_decoder.py
• Configuration: configs/*.yaml + utils/model_setup.py
• Data handling: dataset.py, scripts/extract_images*.py
• Core model: transformer_flow.py
• VAE & losses: misc/wan_vae2.py, misc/ae_losses.py, misc/lpips.py, misc/discriminator.py
• Utilities: utils/{common.py, inference.py, training.py}
• Scripts for automation: scripts/*.sh, scripts/*.py
• Checkpoint & output storage: ckpts/ (external), assets/ (static), any generated outputs.
README insights
• Models: STARFlow (3B parameters) & STARFlow-V (7B parameters)
• Components: T5-XL text encoder, SD-VAE / WAN2.2-VAE, autoregressive flow blocks
• Training: FSDP, mixed precision, block-wise Jacobi sampling
• Sampling: classifier-free guidance, Jacobi iteration for speed.
a. Main components
• CLI Layer: train.py, sample.py, finetune_decoder.py
• Config Loader: configs/*.yaml → utils/model_setup.py
• Data Layer: dataset.py ← scripts/extract_images*
• Model Layer:
· Text Encoder (T5-XL)
· VAE Encoder/Decoder (SD-VAE or WAN2.2-VAE)
· Autoregressive Flow (transformer_flow.py)
• Engine Layer:
· Training engine (utils/training.py) + WandB
· Inference engine (utils/inference.py)
• Storage Layer: ckpts/ for model weights, assets/ & output folders
b. Relationships & interactions
• train.py → model_setup loads config → dataset.py provides batches →
Text Encoder + VAE encode → transformer_flow computes flow →
utils/training orchestrates optimizer, FSDP, loss functions →
checkpoints saved to ckpts/, metrics to WandB.
• sample.py → model_setup loads config & checkpoint →
caption input → Text Encoder + (optional image → VAE encode) →
transformer_flow autoregressive sampling → VAE decode → output media →
utils/inference handles saving and evaluation.
c. Architectural patterns & design principles
• Modular layers: clear separation of CLI, config, data, model, and engines
• Configuration-driven architecture via YAML + model_setup
• Pluggable backbones: easy to swap text encoder or VAE
• Distributed training support (PyTorch FSDP, torchrun)
• Utility modules isolate common code, inference, and training logic.
d. Key technologies & frameworks
• Python & PyTorch (torchrun, FSDP, mixed precision)
• T5-XL (from HuggingFace or internal library)
• SD-VAE and WAN2.2-VAE implementations
• YAML for config, WandB for logging
• Shell scripts for environment & examples.