How zai-org/scail-2 works
1. What this repo is
- SCAIL-2 is primarily an inference repository for controlled character animation and character replacement.
- The core output is a generated video conditioned on a reference image, a reference mask, a driving video or pose video, a per-frame control mask, and a text prompt.
2. Top-level pipeline boundary
- The architecture has a clear split between:
- Input preparation: handled by the external SCAIL-Pose submodule.
- Inference orchestration: generate.py drives model loading and sampling.
- Model/runtime code: the wan/ package contains configs, model modules, samplers, utilities, and distributed wrappers.
- Optional prompt rewriting: prompt_enhancer.py uses Gemini to expand short replacement instructions into better prompts.
3. Data flow from inputs to output
- The main inference path consumes four local artifacts:
- reference image
- reference mask
- driving/pose video
- per-frame mask video
- generate.py validates these paths, loads the media, packages them into model inputs, runs sampling, and writes the final MP4.
- For multi-reference cases, additional image/mask pairs are folded into the same conditioning path.
4. Input preparation subsystem
- SCAIL-Pose is not part of the core model runtime, but it is architecturally important because it generates the conditioning data the model expects.
- It performs pose extraction, pose rendering, and mask generation for both:
- animation mode: end-to-end driving or pose-driven driving
- replacement mode: region replacement with a replacement mask
- This creates the repo’s main boundary: preprocessing in one environment, inference in another.
5. Model family and configuration layer
- wan/configs/ defines model variants and inference presets:
- scail_config_1_3B.py
- scail_config_14B.py
- base Wan I2V/T2V configs
- This suggests the repo supports multiple backbone scales and shares architecture with Wan 2.x-style image/video generation models.
- The config layer is where architecture choices such as resolution support, conditioning layout, and model size are selected.
6. Core model implementation
- wan/modules/ contains the main neural building blocks:
- attention
- CLIP / T5 / XLM-RoBERTa token and text encoders
- VAE for latent video/image representation
- model definitions including model_scail.py and model_scail2.py
- The model stack appears to combine:
- visual latent encoding/decoding
- text conditioning
- control/mask conditioning
- motion transfer logic specific to SCAIL-2
7. Conditioning and control semantics
- SCAIL-2’s architecture is built around a Unified Motion Transfer Interface.
- The conditioning inputs are not just text and reference image; they also encode mask semantics that distinguish background visibility from subject correspondence.
- This is central to the model boundary: the mask video is not incidental metadata, it is a primary control channel.
8. Sampling and generation runtime
- wan/utils/fm_solvers.py and wan/utils/fm_solvers_unipc.py indicate a flow-matching / diffusion-like sampling loop with multiple solver choices.
- generate.py exposes runtime knobs that materially affect output:
- sampling steps
- scheduler shift
- guidance scale
- solver selection
- model offloading for memory management
- The output of the sampler is decoded back into a video file.
9. Distributed and memory-management support
- wan/distributed/ contains FSDP, sequence parallel, Ulysses, and xDiT context-parallel utilities.
- Even though the README focuses on single-GPU inference, these modules show the codebase inherits a distributed training/inference infrastructure from the Wan/SCAIL family.
- For architecture diagrams, these are best treated as optional execution backends, not the main product path.
10. Weights, checkpoint conversion, and deployment format
- Model weights are downloaded as a Hugging Face checkpoint bundle and can be converted with convert.py into a .safetensors file for the wan branch workflow.
- The repo therefore has a deployment boundary between:
- vendor-style packaged checkpoints under ckpt_dir
- runtime-loaded weights referenced by --scail_path
- This conversion step is part of the operational setup, not the model itself.
11. Prompt enhancement auxiliary flow
- prompt_enhancer.py is an optional helper for replacement tasks.
- It samples frames from the driving video, reads the reference image, uses few-shot examples, and calls Gemini to produce a longer prompt suitable for generation.
- This is a separate AI-assisted preprocessing path that feeds back into generate.py.
12. Repository structure summary for a diagram
- A good architecture diagram would show:
- external media inputs and model weights on the left
- SCAIL-Pose preprocessing producing pose videos and masks
- generate.py as the orchestrator
- wan/ as the core model/runtime library
- optional prompt_enhancer.py as a prompt-generation sidecar
- MP4 output on the right
- The dominant system interaction is conditioned video synthesis, with preprocessing and checkpoint management as the main supporting subsystems.
Main entry points: generate.py Generate · model_scail.py SCAIL model · model_scail2.py SCAIL-2 model · fm_solvers.py Samplers
Components 16
Inputs & prep
External media, preprocessing, and prompt assistance
- Media inputs · Reference image, reference mask, driving video, and per-frame control mask/video
- SCAIL-Pose · Pose extraction, rendering, and mask generation for animation and replacement modes
SCAIL-Pose - Prompt enhancer · Optional prompt expansion for replacement tasks using sampled frames and examples
prompt_enhancer.py
Inference runtime
Orchestration and generation entrypoints
- Generate · Validates inputs, loads media, builds model inputs, runs sampling, and writes the output video
generate.py - MP4 output · Final generated video
Model stack
Configs, modules, and samplers
- 1.3B config · SCAIL 1.3B inference preset and variant selection
wan/configs/scail_config_1_3B.py - 14B config · SCAIL 14B inference preset and variant selection
wan/configs/scail_config_14B.py - Shared config · Common configuration pieces reused across Wan and SCAIL variants
wan/configs/shared_config.py - SCAIL model · Main conditioned video generation and motion-transfer model
wan/modules/model_scail.py - SCAIL-2 model · Second-generation SCAIL model definition
wan/modules/model_scail2.py - Core modules · Attention, VAE, CLIP, T5, XLM-R, tokenizers, and model plumbing
wan/modules - Samplers · Flow-matching and UniPC solver loops used during generation
wan/utils/fm_solvers.py - Distributed backends · FSDP, sequence parallel, Ulysses, and xDiT context-parallel utilities
wan/distributed - SCAIL utils · Mask, conditioning, and generation utilities supporting the main pipeline
wan/utils/scail_utils.py
Deployment tools
Checkpoint conversion and packaged weights
- Checkpoint bundle · Downloaded Hugging Face-style weights used by runtime loading
- Convert weights · Converts packaged checkpoints into runtime safetensors format
convert.py
Connections 21
- Media inputs → SCAIL-Pose: prep inputs · Pose and mask preprocessing for animation and replacement
- SCAIL-Pose → Generate: pose/mask video · Provides driving pose streams and control masks
- Prompt enhancer → Generate: prompt · Expanded replacement prompt feeds generation
- Media inputs → Generate: reference media · Reference image and reference mask are loaded directly by inference
- Generate → 1.3B config: selects · Chooses model size and preset
- Generate → 14B config: selects · Chooses model size and preset
- 1.3B config → Shared config: inherits · Builds on shared configuration pieces
- 14B config → Shared config: inherits · Builds on shared configuration pieces
- Generate → SCAIL model: loads · Main inference path uses the SCAIL model implementation
- Generate → SCAIL-2 model: loads · Alternate/next-gen model definition
- SCAIL model → Core modules: uses · Relies on encoders, VAE, and attention layers
- SCAIL-2 model → Core modules: uses · Relies on encoders, VAE, and attention layers
- SCAIL model → SCAIL utils: uses · Conditioning and masking helpers
- SCAIL-2 model → SCAIL utils: uses · Conditioning and masking helpers
- Generate → Samplers: samples with · Runs flow-matching or UniPC solver loops
- Samplers → SCAIL model: iterates · Produces latent video samples from the model
- Samplers → SCAIL-2 model: iterates · Produces latent video samples from the model
- Generate → Distributed backends: optional backend · Optional distributed and memory-parallel execution support
- Convert weights → Checkpoint bundle: converts · Transforms packaged checkpoints into runtime weights
- Generate → Checkpoint bundle: loads · Runtime loads checkpointed model weights
- Generate → MP4 output: writes · Decoded result is saved as MP4