# goatwu/causvid-plus architecture

> Architecture diagram of the GitHub repository goatwu/causvid-plus, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/goatwu/causvid-plus
- Repository: https://github.com/goatwu/causvid-plus

## Overview

##### Project type and purpose
- This is a machine‐learning research & development repository that extends the CausVid video‐generation framework with new models (Wan2.1-T2V-14B), large‐scale data pipelines (50K prompt–video pairs), training (ODE pretraining & DMD distillation), inference (autoregressive & bidirectional long videos), and evaluation (COCO/fid metrics).
- There is no web frontend; instead, it is a CLI-driven, config-driven Python package structured around data preparation, model training, inference, and evaluation.

##### File structure
Top-level:
- causvid/         ← core Python package (data loaders, pipelines, models, inference, evaluation, training scripts)
- configs/         ← YAML files driving all training & inference workflows
- distillation_data/← scripts to download, slice, preprocess MixKit videos
- minimal_inference/← lightweight CLI examples for inference
- sample_dataset/  ← small example prompts/videos for quick tests
- ode_data/        ← scripts to build LMDB datasets of VAE latents & ODE pairs
- utils/           ← utility checks & converters
- tests/           ← unit tests validating each module
- requirements.txt, setup.py, docs/, LICENSE, etc.

Inside causvid/:
- data.py, util.py, scheduler.py, loss.py ← core utilities
- bidirectional_trajectory_pipeline.py, ode_regression.py ← algorithm orchestration
- models/
– sdxl/ wrappers for SDXL model
– wan/ and wan/wan_base/ ← hierarchy of Wan2.1 text2video & image2video architectures, configs, distributed training modules
- evaluation/ ← COCO caption & FID evaluation code

##### README for additional insights
- The README describes environment setup, how to download WAN base models, dataset prep steps (MixKit), dataset conversion → LMDB → ODE pair generation, training commands (torchrun multi-node), and inference examples.
- It mentions WandB for logging (configured via YAML).
- External dependencies: HuggingFace datasets CLI, WAN2.1 checkpoints, MixKit videos, PyTorch FSDP.

##### Components
- Data Ingestion & Preprocessing
– distillation_data scripts
– sample_dataset input
- Latent & ODE Dataset Builder
– causvid/ode_data/create_lmdb_iterative.py
– generate_ode_pairs.py
- Model Training
– ODE Pretraining (train_ode.py)
– DMD Distillation (train_distillation.py)
– Scheduler & distributed strategy (causvid/scheduler.py, FSDP modules)
– WandB logging service (external)
- Model Artifacts
– checkpoint storage (folder or model registry)
- Inference Pipelines
– minimal_inference/*.py (autoregressive, longvideo, bidirectional)
– causvid/bidirectional_trajectory_pipeline.py
- Evaluation
– COCO evaluator & CleanFID (causvid/evaluation/)
– output metrics & reports
- Configuration & CLI
– configs/*.yaml, command-line wrappers

- Raw MixKit videos & prompt JSON → distillation_data → sliced & resized videos → VAE latents
- VAE latents → LMDB (ode_data) → ODE pairs → second LMDB
- LMDB datasets + WAN/CausVid model code + configs → distributed training cluster → model checkpoints
- Checkpoints + configs + prompt files → inference scripts → generated videos
- Generated videos + ground-truth captions → evaluation scripts → metrics

c. Architectural patterns & principles
- Modular Python package: separation into data, models, training, inference, evaluation
- Config-driven workflows: all hyperparams & paths in YAML under configs/
- CLI entrypoints: single-responsibility scripts for each stage
- Distributed training: torchrun + FSDP / xdit context parallel modules
- Batch data pipelines with LMDB for high throughput

d. Key technologies & libraries
- Python 3.10, PyTorch, FSDP (torch.distributed, c10d)
- LMDB for dataset storage
- WAN2.1 model suite, SDXL wrapper
- HuggingFace datasets CLI, WandB for logging
- CleanFID, COCO eval for metrics

##### Detail
- Major Python script or module grouping as a separate component.
- Link configs/*.yaml to every stage that consumes configuration.
- Distributed strategy modules under the training block.
- Evaluation submodules (COCO eval vs. CleanFID) if space allows.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        DistillationScripts["Distillation Scripts"]:::data
        SampleDataset["Sample Dataset"]:::data
    end
    DistillationScripts -->|sliced & resized videos| VAE_LMDB[(VAE Latent LMDB)]:::data
    SampleDataset -->|example prompts/videos| VAE_LMDB

    %% Latent & ODE Dataset Builder
    subgraph "Latent & ODE Dataset Builder"
        CreateLMDB["create_lmdb_iterative.py"]:::data
        ODEPairLMDB[(ODE Pair LMDB)]:::data
        GenerateODE["generate_ode_pairs.py"]:::data
        ODERegression["ODE Regression"]:::data
    end
    VAE_LMDB --> CreateLMDB
    CreateLMDB --> ODEPairLMDB
    ODEPairLMDB --> GenerateODE
    GenerateODE --> ODEPairLMDB
    ODEPairLMDB --> ODERegression

    %% Model Training
    subgraph "Model Training"
        Configs["Configs (YAML)"]:::data
        WANBase["WAN Base Models"]:::compute
        SDXLWrapper["SDXL Wrapper"]:::compute
        TrainODE["train_ode.py"]:::compute
        TrainDistil["train_distillation.py"]:::compute
        Scheduler["scheduler.py"]:::compute
        FSDP["fsdp.py"]:::compute
        XDIT["xdit_context_parallel.py"]:::compute
        Cluster["Distributed Training Cluster"]:::compute
        Checkpoints["Model Checkpoints"]:::data
        WandB["WandB Logging"]:::external
        HuggingFace["HF Datasets CLI"]:::external
    end
    ODEPairLMDB --> TrainODE
    Configs --> TrainODE
    WANBase --> TrainODE
    SDXLWrapper --> TrainODE
    Scheduler --> TrainODE
    FSDP --> TrainODE
    XDIT --> TrainODE
    TrainODE --> Cluster
    TrainDistil --> Cluster
    Cluster --> Checkpoints
    TrainODE --> WandB
    TrainDistil --> WandB
    DistillationScripts --> HuggingFace

    %% Inference Pipelines
    subgraph "Inference"
        Configs --> MinimalInfDir
        Configs --> BiPipeline
        MinimalInfDir["minimal_inference/"]:::inference
        BiPipeline["bidirectional_trajectory_pipeline.py"]:::inference
        CausalInf["causal_inference.py"]:::inference
        BiInf["bidirectional_inference.py"]:::inference
        GenVideos["Generated Videos"]:::inference
    end
    Checkpoints --> MinimalInfDir
    Checkpoints --> BiPipeline
    MinimalInfDir --> GenVideos
    BiPipeline --> CausalInf
    BiPipeline --> BiInf
    CausalInf --> GenVideos
    BiInf --> GenVideos

    %% Evaluation
    subgraph "Evaluation"
        COCoeval["coco_evaluator.py"]:::inference
        CleanFID["CleanFID"]:::inference
        SDXLEval["eval_sdxl_coco.py"]:::inference
        SDXLInferEval["inference_sdxl.py"]:::inference
        Metrics["Evaluation Metrics"]:::inference
    end
    GenVideos --> COCoeval
    GenVideos --> SDXLEval
    GenVideos --> SDXLInferEval
    COCoeval --> Metrics
    CleanFID --> Metrics
    SDXLEval --> Metrics
    SDXLInferEval --> Metrics

    %% Utilities & Configs
    subgraph "Utilities & Helpers"
        DataUtil["data.py"]:::data
        LossUtil["loss.py"]:::data
        GenUtil["util.py"]:::data
        UtilsDir["utils/"]:::data
    end

    %% Tests
    subgraph "Testing"
        TestsWAN["tests/wan/"]:::data
        TestsSDXL["tests/sdxl/"]:::data
        TestsCausal["tests/causal_wan/"]:::data
    end

    %% Click Events
    click DistillationScripts "https://github.com/goatwu/causvid-plus/tree/master/distillation_data/"
    click SampleDataset "https://github.com/goatwu/causvid-plus/tree/master/sample_dataset/"
    click CreateLMDB "https://github.com/goatwu/causvid-plus/blob/master/causvid/ode_data/create_lmdb_iterative.py"
    click GenerateODE "https://github.com/goatwu/causvid-plus/blob/master/causvid/models/wan/generate_ode_pairs.py"
    click ODERegression "https://github.com/goatwu/causvid-plus/blob/master/causvid/ode_regression.py"
    click TrainODE "https://github.com/goatwu/causvid-plus/blob/master/causvid/train_ode.py"
    click TrainDistil "https://github.com/goatwu/causvid-plus/blob/master/causvid/train_distillation.py"
    click Scheduler "https://github.com/goatwu/causvid-plus/blob/master/causvid/scheduler.py"
    click FSDP "https://github.com/goatwu/causvid-plus/blob/master/causvid/models/wan/wan_base/distributed/fsdp.py"
    click XDIT "https://github.com/goatwu/causvid-plus/blob/master/causvid/models/wan/wan_base/distributed/xdit_context_parallel.py"
    click WANBase "https://github.com/goatwu/causvid-plus/tree/master/causvid/models/wan/wan_base/"
    click SDXLWrapper "https://github.com/goatwu/causvid-plus/blob/master/causvid/models/sdxl/sdxl_wrapper.py"
    click MinimalInfDir "https://github.com/goatwu/causvid-plus/tree/master/minimal_inference/"
    click BiPipeline "https://github.com/goatwu/causvid-plus/blob/master/causvid/bidirectional_trajectory_pipeline.py"
    click CausalInf "https://github.com/goatwu/causvid-plus/blob/master/causvid/models/wan/causal_inference.py"
    click BiInf "https://github.com/goatwu/causvid-plus/blob/master/causvid/models/wan/bidirectional_inference.py"
    click COCoeval "https://github.com/goatwu/causvid-plus/blob/master/causvid/evaluation/coco_eval/coco_evaluator.py"
    click CleanFID "https://github.com/goatwu/causvid-plus/tree/master/causvid/evaluation/coco_eval/cleanfid/"
    click SDXLEval "https://github.com/goatwu/causvid-plus/blob/master/causvid/evaluation/eval_sdxl_coco.py"
    click SDXLInferEval "https://github.com/goatwu/causvid-plus/blob/master/causvid/evaluation/inference_sdxl.py"
    click Configs "https://github.com/goatwu/causvid-plus/tree/master/configs/"

    %% Styles
    classDef data fill:#bbdefb,stroke:#0d47a1,color:#0d47a1;
    classDef compute fill:#c8e6c9,stroke:#1b5e20,color:#1b5e20;
    classDef inference fill:#e1bee7,stroke:#4a148c,color:#4a148c;
    classDef external fill:#ffe0b2,stroke:#e65100,color:#e65100;
```
