How goatwu/causvid-plus works
Project type and purpose
• This is a machine‐learning research & development repository that extends the CausVid video‐generation framework with new models (Wan2.1-T2V-14B), large‐scale data pipelines (50K prompt–video pairs), training (ODE pretraining & DMD distillation), inference (autoregressive & bidirectional long videos), and evaluation (COCO/fid metrics).
• There is no web frontend; instead, it is a CLI-driven, config-driven Python package structured around data preparation, model training, inference, and evaluation.
File structure
Top-level:
• causvid/ ← core Python package (data loaders, pipelines, models, inference, evaluation, training scripts)
• configs/ ← YAML files driving all training & inference workflows
• distillation_data/← scripts to download, slice, preprocess MixKit videos
• minimal_inference/← lightweight CLI examples for inference
• sample_dataset/ ← small example prompts/videos for quick tests
• ode_data/ ← scripts to build LMDB datasets of VAE latents & ODE pairs
• utils/ ← utility checks & converters
• tests/ ← unit tests validating each module
• requirements.txt, setup.py, docs/, LICENSE, etc.
Inside causvid/:
• data.py, util.py, scheduler.py, loss.py ← core utilities
• bidirectional_trajectory_pipeline.py, ode_regression.py ← algorithm orchestration
• models/
– sdxl/ wrappers for SDXL model
– wan/ and wan/wan_base/ ← hierarchy of Wan2.1 text2video & image2video architectures, configs, distributed training modules
• evaluation/ ← COCO caption & FID evaluation code
README for additional insights
• The README describes environment setup, how to download WAN base models, dataset prep steps (MixKit), dataset conversion → LMDB → ODE pair generation, training commands (torchrun multi-node), and inference examples.
• It mentions WandB for logging (configured via YAML).
• External dependencies: HuggingFace datasets CLI, WAN2.1 checkpoints, MixKit videos, PyTorch FSDP.
Components
• Data Ingestion & Preprocessing
– distillation_data scripts
– sample_dataset input
• Latent & ODE Dataset Builder
– causvid/ode_data/create_lmdb_iterative.py
– generate_ode_pairs.py
• Model Training
– ODE Pretraining (train_ode.py)
– DMD Distillation (train_distillation.py)
– Scheduler & distributed strategy (causvid/scheduler.py, FSDP modules)
– WandB logging service (external)
• Model Artifacts
– checkpoint storage (folder or model registry)
• Inference Pipelines
– minimal_inference/*.py (autoregressive, longvideo, bidirectional)
– causvid/bidirectional_trajectory_pipeline.py
• Evaluation
– COCO evaluator & CleanFID (causvid/evaluation/)
– output metrics & reports
• Configuration & CLI
– configs/*.yaml, command-line wrappers
• Raw MixKit videos & prompt JSON → distillation_data → sliced & resized videos → VAE latents
• VAE latents → LMDB (ode_data) → ODE pairs → second LMDB
• LMDB datasets + WAN/CausVid model code + configs → distributed training cluster → model checkpoints
• Checkpoints + configs + prompt files → inference scripts → generated videos
• Generated videos + ground-truth captions → evaluation scripts → metrics
c. Architectural patterns & principles
• Modular Python package: separation into data, models, training, inference, evaluation
• Config-driven workflows: all hyperparams & paths in YAML under configs/
• CLI entrypoints: single-responsibility scripts for each stage
• Distributed training: torchrun + FSDP / xdit context parallel modules
• Batch data pipelines with LMDB for high throughput
d. Key technologies & libraries
• Python 3.10, PyTorch, FSDP (torch.distributed, c10d)
• LMDB for dataset storage
• WAN2.1 model suite, SDXL wrapper
• HuggingFace datasets CLI, WandB for logging
• CleanFID, COCO eval for metrics
Detail
• Major Python script or module grouping as a separate component.
• Link configs/*.yaml to every stage that consumes configuration.
• Distributed strategy modules under the training block.
• Evaluation submodules (COCO eval vs. CleanFID) if space allows.