# zai-org/scail-2 architecture

> Architecture diagram of the GitHub repository zai-org/scail-2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zai-org/scail-2
- Repository: https://github.com/zai-org/scail-2

## Overview

1. **What this repo is**
   - SCAIL-2 is primarily an **inference repository** for controlled character animation and character replacement.
   - The core output is a **generated video** conditioned on a reference image, a reference mask, a driving video or pose video, a per-frame control mask, and a text prompt.

2. **Top-level pipeline boundary**
   - The architecture has a clear split between:
     - **Input preparation**: handled by the external `SCAIL-Pose` submodule.
     - **Inference orchestration**: `generate.py` drives model loading and sampling.
     - **Model/runtime code**: the `wan/` package contains configs, model modules, samplers, utilities, and distributed wrappers.
     - **Optional prompt rewriting**: `prompt_enhancer.py` uses Gemini to expand short replacement instructions into better prompts.

3. **Data flow from inputs to output**
   - The main inference path consumes four local artifacts:
     - reference image
     - reference mask
     - driving/pose video
     - per-frame mask video
   - `generate.py` validates these paths, loads the media, packages them into model inputs, runs sampling, and writes the final MP4.
   - For multi-reference cases, additional image/mask pairs are folded into the same conditioning path.

4. **Input preparation subsystem**
   - `SCAIL-Pose` is not part of the core model runtime, but it is architecturally important because it generates the conditioning data the model expects.
   - It performs pose extraction, pose rendering, and mask generation for both:
     - **animation mode**: end-to-end driving or pose-driven driving
     - **replacement mode**: region replacement with a replacement mask
   - This creates the repo’s main boundary: **preprocessing in one environment, inference in another**.

5. **Model family and configuration layer**
   - `wan/configs/` defines model variants and inference presets:
     - `scail_config_1_3B.py`
     - `scail_config_14B.py`
     - base Wan I2V/T2V configs
   - This suggests the repo supports multiple backbone scales and shares architecture with Wan 2.x-style image/video generation models.
   - The config layer is where architecture choices such as resolution support, conditioning layout, and model size are selected.

6. **Core model implementation**
   - `wan/modules/` contains the main neural building blocks:
     - attention
     - CLIP / T5 / XLM-RoBERTa token and text encoders
     - VAE for latent video/image representation
     - model definitions including `model_scail.py` and `model_scail2.py`
   - The model stack appears to combine:
     - visual latent encoding/decoding
     - text conditioning
     - control/mask conditioning
     - motion transfer logic specific to SCAIL-2

7. **Conditioning and control semantics**
   - SCAIL-2’s architecture is built around a **Unified Motion Transfer Interface**.
   - The conditioning inputs are not just text and reference image; they also encode **mask semantics** that distinguish background visibility from subject correspondence.
   - This is central to the model boundary: the mask video is not incidental metadata, it is a primary control channel.

8. **Sampling and generation runtime**
   - `wan/utils/fm_solvers.py` and `wan/utils/fm_solvers_unipc.py` indicate a **flow-matching / diffusion-like sampling loop** with multiple solver choices.
   - `generate.py` exposes runtime knobs that materially affect output:
     - sampling steps
     - scheduler shift
     - guidance scale
     - solver selection
     - model offloading for memory management
   - The output of the sampler is decoded back into a video file.

9. **Distributed and memory-management support**
   - `wan/distributed/` contains FSDP, sequence parallel, Ulysses, and xDiT context-parallel utilities.
   - Even though the README focuses on single-GPU inference, these modules show the codebase inherits a **distributed training/inference infrastructure** from the Wan/SCAIL family.
   - For architecture diagrams, these are best treated as **optional execution backends**, not the main product path.

10. **Weights, checkpoint conversion, and deployment format**
   - Model weights are downloaded as a Hugging Face checkpoint bundle and can be converted with `convert.py` into a `.safetensors` file for the `wan` branch workflow.
   - The repo therefore has a deployment boundary between:
     - vendor-style packaged checkpoints under `ckpt_dir`
     - runtime-loaded weights referenced by `--scail_path`
   - This conversion step is part of the operational setup, not the model itself.

11. **Prompt enhancement auxiliary flow**
   - `prompt_enhancer.py` is an optional helper for replacement tasks.
   - It samples frames from the driving video, reads the reference image, uses few-shot examples, and calls Gemini to produce a longer prompt suitable for generation.
   - This is a separate AI-assisted preprocessing path that feeds back into `generate.py`.

12. **Repository structure summary for a diagram**
   - A good architecture diagram would show:
     - external media inputs and model weights on the left
     - `SCAIL-Pose` preprocessing producing pose videos and masks
     - `generate.py` as the orchestrator
     - `wan/` as the core model/runtime library
     - optional `prompt_enhancer.py` as a prompt-generation sidecar
     - MP4 output on the right
   - The dominant system interaction is **conditioned video synthesis**, with preprocessing and checkpoint management as the main supporting subsystems.

Main entry points: [`generate.py`](https://github.com/zai-org/scail-2/blob/wan-scail2/generate.py) (Generate), [`wan/modules/model_scail.py`](https://github.com/zai-org/scail-2/blob/wan-scail2/wan/modules/model_scail.py) (SCAIL model), [`wan/modules/model_scail2.py`](https://github.com/zai-org/scail-2/blob/wan-scail2/wan/modules/model_scail2.py) (SCAIL-2 model), [`wan/utils/fm_solvers.py`](https://github.com/zai-org/scail-2/blob/wan-scail2/wan/utils/fm_solvers.py) (Samplers)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g_inputs["Inputs & prep"]
  node_external_media["Media inputs<br/>artifact"]
  node_scail_pose["SCAIL-Pose<br/>preprocess submodule"]
  node_prompt_enhancer["Prompt enhancer<br/>gemini helper<br/>[prompt_enhancer.py]"]
end

subgraph group_g_runtime["Inference runtime"]
  node_generate_py["Generate<br/>inference orchestrator<br/>[generate.py]"]
  node_output_video["MP4 output<br/>artifact"]
end

subgraph group_g_model["Model stack"]
  node_scail_config_1_3b["1.3B config<br/>model config"]
  node_scail_config_14b["14B config<br/>model config"]
  node_shared_config["Shared config<br/>config base<br/>[shared_config.py]"]
  node_model_scail["SCAIL model<br/>video model<br/>[model_scail.py]"]
  node_model_scail2["SCAIL-2 model<br/>video model<br/>[model_scail2.py]"]
  node_attention_vae_text["Core modules<br/>layers & encoders"]
  node_samplers["Samplers<br/>solver runtime<br/>[fm_solvers.py]"]
  node_dist_backends["Distributed backends<br/>optional runtime"]
  node_scail_utils["SCAIL utils<br/>runtime helpers<br/>[scail_utils.py]"]
end

subgraph group_g_ops["Deployment tools"]
  node_weights_bundle[("Checkpoint bundle<br/>model weights")]
  node_convert_py["Convert weights<br/>conversion tool<br/>[convert.py]"]
end

node_external_media -->|"prep inputs"| node_scail_pose
node_scail_pose -->|"pose/mask video"| node_generate_py
node_prompt_enhancer -->|"prompt"| node_generate_py
node_external_media -->|"reference media"| node_generate_py
node_generate_py -->|"selects"| node_scail_config_1_3b
node_generate_py -->|"selects"| node_scail_config_14b
node_scail_config_1_3b -->|"inherits"| node_shared_config
node_scail_config_14b -->|"inherits"| node_shared_config
node_generate_py -->|"loads"| node_model_scail
node_generate_py -->|"loads"| node_model_scail2
node_model_scail -->|"uses"| node_attention_vae_text
node_model_scail2 -->|"uses"| node_attention_vae_text
node_model_scail -->|"uses"| node_scail_utils
node_model_scail2 -->|"uses"| node_scail_utils
node_generate_py -->|"samples with"| node_samplers
node_samplers -->|"iterates"| node_model_scail
node_samplers -->|"iterates"| node_model_scail2
node_generate_py -.->|"optional backend"| node_dist_backends
node_convert_py -->|"converts"| node_weights_bundle
node_generate_py -->|"loads"| node_weights_bundle
node_generate_py -->|"writes"| node_output_video

click node_scail_pose "https://github.com/zai-org/scail-2/tree/wan-scail2/SCAIL-Pose"
click node_prompt_enhancer "https://github.com/zai-org/scail-2/blob/wan-scail2/prompt_enhancer.py"
click node_generate_py "https://github.com/zai-org/scail-2/blob/wan-scail2/generate.py"
click node_scail_config_1_3b "https://github.com/zai-org/scail-2/blob/wan-scail2/wan/configs/scail_config_1_3B.py"
click node_scail_config_14b "https://github.com/zai-org/scail-2/blob/wan-scail2/wan/configs/scail_config_14B.py"
click node_shared_config "https://github.com/zai-org/scail-2/blob/wan-scail2/wan/configs/shared_config.py"
click node_model_scail "https://github.com/zai-org/scail-2/blob/wan-scail2/wan/modules/model_scail.py"
click node_model_scail2 "https://github.com/zai-org/scail-2/blob/wan-scail2/wan/modules/model_scail2.py"
click node_attention_vae_text "https://github.com/zai-org/scail-2/tree/wan-scail2/wan/modules"
click node_samplers "https://github.com/zai-org/scail-2/blob/wan-scail2/wan/utils/fm_solvers.py"
click node_dist_backends "https://github.com/zai-org/scail-2/tree/wan-scail2/wan/distributed"
click node_scail_utils "https://github.com/zai-org/scail-2/blob/wan-scail2/wan/utils/scail_utils.py"
click node_convert_py "https://github.com/zai-org/scail-2/blob/wan-scail2/convert.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_external_media,node_scail_pose,node_prompt_enhancer toneBlue
class node_generate_py,node_output_video toneAmber
class node_scail_config_1_3b,node_scail_config_14b,node_shared_config,node_model_scail,node_model_scail2,node_attention_vae_text,node_samplers,node_dist_backends,node_scail_utils toneMint
class node_weights_bundle,node_convert_py toneRose
```

## Components

### Inputs & prep

External media, preprocessing, and prompt assistance

- **Media inputs**: artifact · Reference image, reference mask, driving video, and per-frame control mask/video
- [SCAIL-Pose](https://github.com/zai-org/scail-2/tree/wan-scail2/SCAIL-Pose): preprocess submodule · Pose extraction, rendering, and mask generation for animation and replacement modes · `SCAIL-Pose`
- [Prompt enhancer](https://github.com/zai-org/scail-2/blob/wan-scail2/prompt_enhancer.py): gemini helper · Optional prompt expansion for replacement tasks using sampled frames and examples · `prompt_enhancer.py`

### Inference runtime

Orchestration and generation entrypoints

- [Generate](https://github.com/zai-org/scail-2/blob/wan-scail2/generate.py): inference orchestrator · Validates inputs, loads media, builds model inputs, runs sampling, and writes the output video · `generate.py`
- **MP4 output**: artifact · Final generated video

### Model stack

Configs, modules, and samplers

- [1.3B config](https://github.com/zai-org/scail-2/blob/wan-scail2/wan/configs/scail_config_1_3B.py): model config · SCAIL 1.3B inference preset and variant selection · `wan/configs/scail_config_1_3B.py`
- [14B config](https://github.com/zai-org/scail-2/blob/wan-scail2/wan/configs/scail_config_14B.py): model config · SCAIL 14B inference preset and variant selection · `wan/configs/scail_config_14B.py`
- [Shared config](https://github.com/zai-org/scail-2/blob/wan-scail2/wan/configs/shared_config.py): config base · Common configuration pieces reused across Wan and SCAIL variants · `wan/configs/shared_config.py`
- [SCAIL model](https://github.com/zai-org/scail-2/blob/wan-scail2/wan/modules/model_scail.py): video model · Main conditioned video generation and motion-transfer model · `wan/modules/model_scail.py`
- [SCAIL-2 model](https://github.com/zai-org/scail-2/blob/wan-scail2/wan/modules/model_scail2.py): video model · Second-generation SCAIL model definition · `wan/modules/model_scail2.py`
- [Core modules](https://github.com/zai-org/scail-2/tree/wan-scail2/wan/modules): layers & encoders · Attention, VAE, CLIP, T5, XLM-R, tokenizers, and model plumbing · `wan/modules`
- [Samplers](https://github.com/zai-org/scail-2/blob/wan-scail2/wan/utils/fm_solvers.py): solver runtime · Flow-matching and UniPC solver loops used during generation · `wan/utils/fm_solvers.py`
- [Distributed backends](https://github.com/zai-org/scail-2/tree/wan-scail2/wan/distributed): optional runtime · FSDP, sequence parallel, Ulysses, and xDiT context-parallel utilities · `wan/distributed`
- [SCAIL utils](https://github.com/zai-org/scail-2/blob/wan-scail2/wan/utils/scail_utils.py): runtime helpers · Mask, conditioning, and generation utilities supporting the main pipeline · `wan/utils/scail_utils.py`

### Deployment tools

Checkpoint conversion and packaged weights

- **Checkpoint bundle**: model weights · Downloaded Hugging Face-style weights used by runtime loading
- [Convert weights](https://github.com/zai-org/scail-2/blob/wan-scail2/convert.py): conversion tool · Converts packaged checkpoints into runtime safetensors format · `convert.py`

## Connections

- Media inputs → SCAIL-Pose: prep inputs · Pose and mask preprocessing for animation and replacement
- SCAIL-Pose → Generate: pose/mask video · Provides driving pose streams and control masks
- Prompt enhancer → Generate: prompt · Expanded replacement prompt feeds generation
- Media inputs → Generate: reference media · Reference image and reference mask are loaded directly by inference
- Generate → 1.3B config: selects · Chooses model size and preset
- Generate → 14B config: selects · Chooses model size and preset
- 1.3B config → Shared config: inherits · Builds on shared configuration pieces
- 14B config → Shared config: inherits · Builds on shared configuration pieces
- Generate → SCAIL model: loads · Main inference path uses the SCAIL model implementation
- Generate → SCAIL-2 model: loads · Alternate/next-gen model definition
- SCAIL model → Core modules: uses · Relies on encoders, VAE, and attention layers
- SCAIL-2 model → Core modules: uses · Relies on encoders, VAE, and attention layers
- SCAIL model → SCAIL utils: uses · Conditioning and masking helpers
- SCAIL-2 model → SCAIL utils: uses · Conditioning and masking helpers
- Generate → Samplers: samples with · Runs flow-matching or UniPC solver loops
- Samplers → SCAIL model: iterates · Produces latent video samples from the model
- Samplers → SCAIL-2 model: iterates · Produces latent video samples from the model
- Generate → Distributed backends: optional backend · Optional distributed and memory-parallel execution support
- Convert weights → Checkpoint bundle: converts · Transforms packaged checkpoints into runtime weights
- Generate → Checkpoint bundle: loads · Runtime loads checkpointed model weights
- Generate → MP4 output: writes · Decoded result is saved as MP4
