# apple-aiml-research/ml-fastframes architecture

> Architecture diagram of the GitHub repository apple-aiml-research/ml-fastframes, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apple-aiml-research/ml-fastframes
- Repository: https://github.com/apple-aiml-research/ml-fastframes

## Overview

FastFrames is a video-generation system offering text-to-video generation, an interactive camera-conditioned LingBot demo, model training, and evaluation/profiling. The main generation flow runs from a prompt through the FastFrames inference pipeline to decoded video. LingBot adds image, prompt, and camera conditioning to chunk-wise generation. Training and evaluation are distinct product workflows. Internal model components and checkpoint loading are shown where the source index or excerpts support their relationships; unsampled demo and training wiring is represented only at the documented subsystem level.

Main entry points: [`fastframes/pipeline.py`](https://github.com/apple-aiml-research/ml-fastframes/blob/main/fastframes/pipeline.py) (FastFrames pipeline), [`apps/lingbot/pipeline.py`](https://github.com/apple-aiml-research/ml-fastframes/blob/main/apps/lingbot/pipeline.py) (LingBot pipeline), [`fastframes/wan/causal_model.py`](https://github.com/apple-aiml-research/ml-fastframes/blob/main/fastframes/wan/causal_model.py) (Causal Wan model), [`fastframes/training`](https://github.com/apple-aiml-research/ml-fastframes/tree/main/fastframes/training) (Training workflows)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_generation["Video generation"]
  node_generate_cli["Generation command<br/>[generate.py]"]
  node_ff_pipeline["FastFrames pipeline<br/>[pipeline.py]"]
  node_video_output["Generated video<br/>[video.py]"]
end

subgraph group_lingbot["LingBot demo"]
  node_demo_ui["Demo interface<br/>[app.js]"]
  node_demo_server["LingBot demo<br/>[demo.py]"]
  node_lingbot_pipeline["LingBot pipeline<br/>[pipeline.py]"]
  node_conditioning["Prompt and camera<br/>[conditioning.py]"]
  node_lingbot_dit["LingBot DiT<br/>[dit.py]"]
  node_lingbot_refiner["LingBot refiner<br/>[refiner.py]"]
  node_lingbot_decoder["LingBot decoder<br/>[decoder.py]"]
end

subgraph group_models["Model components"]
  node_checkpoints["Checkpoint loading<br/>[checkpoints.py]"]
  node_wan_encoder["Text encoder<br/>[text_encoder.py]"]
  node_wan_dit["Causal Wan model<br/>[causal_model.py]"]
  node_kv_refinement["KV refinement<br/>[kv_refinement.py]"]
  node_pixel_decoder["Pixel decoder<br/>[pixel_decoder.py]"]
  node_vae["Wan VAE<br/>[vae.py]"]
end

subgraph group_training["Training"]
  node_train_command["Training command<br/>[train.py]"]
  node_training_system["Training workflows"]
end

subgraph group_evaluation["Evaluation"]
  node_vbench["VBench evaluation<br/>[protocol.py]"]
  node_latency["Latency profiling<br/>[timing.py]"]
end

node_user(("User"))

node_user -->|"sets prompt"| node_generate_cli
node_generate_cli -->|"runs generation"| node_ff_pipeline
node_ff_pipeline -->|"loads weights"| node_checkpoints
node_ff_pipeline -->|"encodes text"| node_wan_encoder
node_ff_pipeline -->|"denoises frames"| node_wan_dit
node_ff_pipeline -->|"refines cache"| node_kv_refinement
node_ff_pipeline -->|"decodes pixels"| node_pixel_decoder
node_ff_pipeline -.->|"writes video"| node_video_output
node_user -->|"controls demo"| node_demo_ui
node_demo_ui -.->|"sends requests"| node_demo_server
node_demo_server -.->|"runs sessions"| node_lingbot_pipeline
node_lingbot_pipeline -->|"builds conditioning"| node_conditioning
node_lingbot_pipeline -->|"denoises chunks"| node_lingbot_dit
node_lingbot_pipeline -->|"refines cache"| node_lingbot_refiner
node_lingbot_pipeline -->|"decodes chunks"| node_lingbot_decoder
node_lingbot_pipeline -->|"loads weights"| node_checkpoints
node_lingbot_dit -->|"uses camera data"| node_conditioning
node_lingbot_refiner -->|"uses DiT features"| node_lingbot_dit
node_lingbot_decoder -->|"uses conditioning"| node_conditioning
node_train_command -.->|"starts training"| node_training_system
node_training_system -->|"trains generator"| node_wan_dit
node_training_system -->|"trains refinement"| node_kv_refinement
node_training_system -->|"trains decoder"| node_pixel_decoder
node_user -->|"requests evaluation"| node_vbench
node_vbench -->|"generates samples"| node_ff_pipeline
node_vbench -->|"checks video contract"| node_video_output
node_user -->|"requests timings"| node_latency
node_latency -->|"profiles DiT"| node_wan_dit
node_latency -->|"profiles decoder"| node_pixel_decoder
node_latency -->|"profiles refinement"| node_kv_refinement
node_wan_dit -.->|"consumes text context"| node_wan_encoder
node_wan_dit -.->|"uses latent model"| node_vae

click node_generate_cli "https://github.com/apple-aiml-research/ml-fastframes/blob/main/scripts/generate.py"
click node_ff_pipeline "https://github.com/apple-aiml-research/ml-fastframes/blob/main/fastframes/pipeline.py"
click node_video_output "https://github.com/apple-aiml-research/ml-fastframes/blob/main/fastframes/video.py"
click node_demo_ui "https://github.com/apple-aiml-research/ml-fastframes/blob/main/apps/lingbot/static/app.js"
click node_demo_server "https://github.com/apple-aiml-research/ml-fastframes/blob/main/apps/lingbot/demo.py"
click node_lingbot_pipeline "https://github.com/apple-aiml-research/ml-fastframes/blob/main/apps/lingbot/pipeline.py"
click node_conditioning "https://github.com/apple-aiml-research/ml-fastframes/blob/main/apps/lingbot/conditioning.py"
click node_lingbot_dit "https://github.com/apple-aiml-research/ml-fastframes/blob/main/apps/lingbot/dit.py"
click node_lingbot_refiner "https://github.com/apple-aiml-research/ml-fastframes/blob/main/apps/lingbot/refiner.py"
click node_lingbot_decoder "https://github.com/apple-aiml-research/ml-fastframes/blob/main/apps/lingbot/decoder.py"
click node_checkpoints "https://github.com/apple-aiml-research/ml-fastframes/blob/main/fastframes/checkpoints.py"
click node_wan_encoder "https://github.com/apple-aiml-research/ml-fastframes/blob/main/fastframes/wan/text_encoder.py"
click node_wan_dit "https://github.com/apple-aiml-research/ml-fastframes/blob/main/fastframes/wan/causal_model.py"
click node_kv_refinement "https://github.com/apple-aiml-research/ml-fastframes/blob/main/fastframes/models/kv_refinement.py"
click node_pixel_decoder "https://github.com/apple-aiml-research/ml-fastframes/blob/main/fastframes/models/pixel_decoder.py"
click node_vae "https://github.com/apple-aiml-research/ml-fastframes/blob/main/fastframes/wan/vae.py"
click node_train_command "https://github.com/apple-aiml-research/ml-fastframes/blob/main/scripts/train.py"
click node_training_system "https://github.com/apple-aiml-research/ml-fastframes/tree/main/fastframes/training"
click node_vbench "https://github.com/apple-aiml-research/ml-fastframes/blob/main/evaluation/vbench/protocol.py"
click node_latency "https://github.com/apple-aiml-research/ml-fastframes/blob/main/evaluation/latency/timing.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_generate_cli,node_ff_pipeline,node_video_output,node_user toneBlue
class node_demo_ui,node_demo_server,node_lingbot_pipeline,node_conditioning,node_lingbot_dit,node_lingbot_refiner,node_lingbot_decoder toneAmber
class node_checkpoints,node_wan_encoder,node_wan_dit,node_kv_refinement,node_pixel_decoder,node_vae toneMint
class node_train_command,node_training_system toneRose
class node_vbench,node_latency toneIndigo
```

## Components

### Video generation

- [Generation command](https://github.com/apple-aiml-research/ml-fastframes/blob/main/scripts/generate.py): `scripts/generate.py`
- [FastFrames pipeline](https://github.com/apple-aiml-research/ml-fastframes/blob/main/fastframes/pipeline.py): `fastframes/pipeline.py`
- [Generated video](https://github.com/apple-aiml-research/ml-fastframes/blob/main/fastframes/video.py): `fastframes/video.py`

### LingBot demo

- [Demo interface](https://github.com/apple-aiml-research/ml-fastframes/blob/main/apps/lingbot/static/app.js): `apps/lingbot/static/app.js`
- [LingBot demo](https://github.com/apple-aiml-research/ml-fastframes/blob/main/apps/lingbot/demo.py): `apps/lingbot/demo.py`
- [LingBot pipeline](https://github.com/apple-aiml-research/ml-fastframes/blob/main/apps/lingbot/pipeline.py): `apps/lingbot/pipeline.py`
- [Prompt and camera](https://github.com/apple-aiml-research/ml-fastframes/blob/main/apps/lingbot/conditioning.py): `apps/lingbot/conditioning.py`
- [LingBot DiT](https://github.com/apple-aiml-research/ml-fastframes/blob/main/apps/lingbot/dit.py): `apps/lingbot/dit.py`
- [LingBot refiner](https://github.com/apple-aiml-research/ml-fastframes/blob/main/apps/lingbot/refiner.py): `apps/lingbot/refiner.py`
- [LingBot decoder](https://github.com/apple-aiml-research/ml-fastframes/blob/main/apps/lingbot/decoder.py): `apps/lingbot/decoder.py`

### Model components

- [Checkpoint loading](https://github.com/apple-aiml-research/ml-fastframes/blob/main/fastframes/checkpoints.py): `fastframes/checkpoints.py`
- [Text encoder](https://github.com/apple-aiml-research/ml-fastframes/blob/main/fastframes/wan/text_encoder.py): `fastframes/wan/text_encoder.py`
- [Causal Wan model](https://github.com/apple-aiml-research/ml-fastframes/blob/main/fastframes/wan/causal_model.py): `fastframes/wan/causal_model.py`
- [KV refinement](https://github.com/apple-aiml-research/ml-fastframes/blob/main/fastframes/models/kv_refinement.py): `fastframes/models/kv_refinement.py`
- [Pixel decoder](https://github.com/apple-aiml-research/ml-fastframes/blob/main/fastframes/models/pixel_decoder.py): `fastframes/models/pixel_decoder.py`
- [Wan VAE](https://github.com/apple-aiml-research/ml-fastframes/blob/main/fastframes/wan/vae.py): `fastframes/wan/vae.py`

### Training

- [Training command](https://github.com/apple-aiml-research/ml-fastframes/blob/main/scripts/train.py): `scripts/train.py`
- [Training workflows](https://github.com/apple-aiml-research/ml-fastframes/tree/main/fastframes/training): `fastframes/training`

### Evaluation

- [VBench evaluation](https://github.com/apple-aiml-research/ml-fastframes/blob/main/evaluation/vbench/protocol.py): `evaluation/vbench/protocol.py`
- [Latency profiling](https://github.com/apple-aiml-research/ml-fastframes/blob/main/evaluation/latency/timing.py): `evaluation/latency/timing.py`

### Other components

- **User**

## Connections

- User → Generation command: sets prompt
- Generation command → FastFrames pipeline: runs generation
- FastFrames pipeline → Checkpoint loading: loads weights (evidence: `fastframes/pipeline.py`)
- FastFrames pipeline → Text encoder: encodes text (evidence: `fastframes/pipeline.py`)
- FastFrames pipeline → Causal Wan model: denoises frames (evidence: `fastframes/pipeline.py`)
- FastFrames pipeline → KV refinement: refines cache (evidence: `fastframes/pipeline.py`)
- FastFrames pipeline → Pixel decoder: decodes pixels (evidence: `fastframes/pipeline.py`)
- FastFrames pipeline → Generated video: writes video
- User → Demo interface: controls demo
- Demo interface → LingBot demo: sends requests
- LingBot demo → LingBot pipeline: runs sessions
- LingBot pipeline → Prompt and camera: builds conditioning (evidence: `apps/lingbot/pipeline.py`)
- LingBot pipeline → LingBot DiT: denoises chunks (evidence: `apps/lingbot/pipeline.py`)
- LingBot pipeline → LingBot refiner: refines cache (evidence: `apps/lingbot/pipeline.py`)
- LingBot pipeline → LingBot decoder: decodes chunks (evidence: `apps/lingbot/pipeline.py`)
- LingBot pipeline → Checkpoint loading: loads weights (evidence: `apps/lingbot/pipeline.py`)
- LingBot DiT → Prompt and camera: uses camera data (evidence: `apps/lingbot/dit.py`)
- LingBot refiner → LingBot DiT: uses DiT features (evidence: `apps/lingbot/refiner.py`)
- LingBot decoder → Prompt and camera: uses conditioning (evidence: `apps/lingbot/decoder.py`)
- Training command → Training workflows: starts training
- Training workflows → Causal Wan model: trains generator (evidence: `fastframes/training/kv_refinement.py`)
- Training workflows → KV refinement: trains refinement (evidence: `fastframes/training/kv_refinement.py`)
- Training workflows → Pixel decoder: trains decoder (evidence: `fastframes/training/pixel_decoder.py`)
- User → VBench evaluation: requests evaluation
- VBench evaluation → FastFrames pipeline: generates samples (evidence: `evaluation/vbench/generate_baselines.py`)
- VBench evaluation → Generated video: checks video contract (evidence: `evaluation/vbench/protocol.py`)
- User → Latency profiling: requests timings
- Latency profiling → Causal Wan model: profiles DiT (evidence: `evaluation/latency/dit.py`)
- Latency profiling → Pixel decoder: profiles decoder (evidence: `evaluation/latency/decoders.py`)
- Latency profiling → KV refinement: profiles refinement
- Causal Wan model → Text encoder: consumes text context
- Causal Wan model → Wan VAE: uses latent model
