How apple-aiml-research/ml-fastframes works
FastFrames is a video-generation system offering text-to-video generation, an interactive camera-conditioned LingBot demo, model training, and evaluation/profiling. The main generation flow runs from a prompt through the FastFrames inference pipeline to decoded video. LingBot adds image, prompt, and camera conditioning to chunk-wise generation. Training and evaluation are distinct product workflows. Internal model components and checkpoint loading are shown where the source index or excerpts support their relationships; unsampled demo and training wiring is represented only at the documented subsystem level.
Main entry points: fastframes/pipeline.py FastFrames pipeline · lingbot/pipeline.py LingBot pipeline · causal_model.py Causal Wan model · training Training workflows
Components 21
Video generation
- Generation command
scripts/generate.py - FastFrames pipeline
fastframes/pipeline.py - Generated video
fastframes/video.py
LingBot demo
- Demo interface
apps/lingbot/static/app.js - LingBot demo
apps/lingbot/demo.py - LingBot pipeline
apps/lingbot/pipeline.py - Prompt and camera
apps/lingbot/conditioning.py - LingBot DiT
apps/lingbot/dit.py - LingBot refiner
apps/lingbot/refiner.py - LingBot decoder
apps/lingbot/decoder.py
Model components
- Checkpoint loading
fastframes/checkpoints.py - Text encoder
fastframes/wan/text_encoder.py - Causal Wan model
fastframes/wan/causal_model.py - KV refinement
fastframes/models/kv_refinement.py - Pixel decoder
fastframes/models/pixel_decoder.py - Wan VAE
fastframes/wan/vae.py
Training
- Training command
scripts/train.py - Training workflows
fastframes/training
Evaluation
- VBench evaluation
evaluation/vbench/protocol.py - Latency profiling
evaluation/latency/timing.py
Other components
- User
Connections 32
- User → Generation command: sets prompt
- Generation command → FastFrames pipeline: runs generation
- FastFrames pipeline → Checkpoint loading: loads weights
fastframes/pipeline.py - FastFrames pipeline → Text encoder: encodes text
fastframes/pipeline.py - FastFrames pipeline → Causal Wan model: denoises frames
fastframes/pipeline.py - FastFrames pipeline → KV refinement: refines cache
fastframes/pipeline.py - FastFrames pipeline → Pixel decoder: decodes pixels
fastframes/pipeline.py - FastFrames pipeline → Generated video: writes video
- User → Demo interface: controls demo
- Demo interface → LingBot demo: sends requests
- LingBot demo → LingBot pipeline: runs sessions
- LingBot pipeline → Prompt and camera: builds conditioning
apps/lingbot/pipeline.py - LingBot pipeline → LingBot DiT: denoises chunks
apps/lingbot/pipeline.py - LingBot pipeline → LingBot refiner: refines cache
apps/lingbot/pipeline.py - LingBot pipeline → LingBot decoder: decodes chunks
apps/lingbot/pipeline.py - LingBot pipeline → Checkpoint loading: loads weights
apps/lingbot/pipeline.py - LingBot DiT → Prompt and camera: uses camera data
apps/lingbot/dit.py - LingBot refiner → LingBot DiT: uses DiT features
apps/lingbot/refiner.py - LingBot decoder → Prompt and camera: uses conditioning
apps/lingbot/decoder.py - Training command → Training workflows: starts training
- Training workflows → Causal Wan model: trains generator
fastframes/training/kv_refinement.py - Training workflows → KV refinement: trains refinement
fastframes/training/kv_refinement.py - Training workflows → Pixel decoder: trains decoder
fastframes/training/pixel_decoder.py - User → VBench evaluation: requests evaluation
- VBench evaluation → FastFrames pipeline: generates samples
evaluation/vbench/generate_baselines.py - VBench evaluation → Generated video: checks video contract
evaluation/vbench/protocol.py - User → Latency profiling: requests timings
- Latency profiling → Causal Wan model: profiles DiT
evaluation/latency/dit.py - Latency profiling → Pixel decoder: profiles decoder
evaluation/latency/decoders.py - Latency profiling → KV refinement: profiles refinement
- Causal Wan model → Text encoder: consumes text context
- Causal Wan model → Wan VAE: uses latent model