# netflix/void-model architecture

> Architecture diagram of the GitHub repository netflix/void-model, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/netflix/void-model
- Repository: https://github.com/netflix/void-model

## Overview

1. **What this repo is**
   VOID is a video object-removal system with two distinct halves: a **mask-generation pipeline** that finds the object and its interaction regions, and a **video inpainting pipeline** that synthesizes the cleaned video. The central architectural idea is the **quadmask**: a 4-valued per-pixel conditioning signal that tells the model what to delete, what to preserve, and what interaction area to reconstruct.

2. **Top-level pipeline boundary**
   The repository is organized around three flows:
   - **Mask creation**: raw video → point selection → SAM2 segmentation → VLM reasoning → quadmask
   - **Inference**: video + quadmask + background prompt → Pass 1 inpainting → optional Pass 2 refinement
   - **Training/data generation**: synthetic paired videos + masks → training metadata → Pass 1/Pass 2 model training  
   This separation is important for architecture diagrams because mask generation, model inference, and synthetic data production are all independent subsystems.

3. **Mask-generation subsystem: `VLM-MASK-REASONER/`**
   This directory implements a four-stage preprocessing pipeline. A human first selects seed points in a GUI, then **SAM2** produces the object mask, a **Gemini/VLM** stage reasons about physically affected regions, a grey-mask stage expands those regions, and a final combiner produces the quadmask video. The output artifact is `quadmask_0.mp4`, which becomes the primary conditioning input for inference.

4. **Interactive tools for mask refinement**
   Two GUIs sit on the mask boundary:
   - `point_selector_gui.py` for initial point annotation
   - `edit_quadmask.py` for manual correction of the generated quadmask  
   These are human-in-the-loop correction points, not part of the learned model, and they exist to improve the quality of the conditioning signal before inference.

5. **Inference subsystem: `inference/`**
   Inference is explicitly two-pass:
   - **Pass 1** uses the base CogVideoX-Fun inpainting model adapted for VOID, conditioned on the quadmask and text prompt.
   - **Pass 2** refines the result using **warped noise** derived from Pass 1, improving temporal consistency over long clips.  
   This two-pass design is the main runtime architecture for output generation.

6. **Core model package: `videox_fun/`**
   This is the shared model/runtime layer used by training, inference, and UI. It contains:
   - **pipelines** for CogVideoX-Fun and inpainting
   - **models** for the transformer, VAE, and cache utilities
   - **data** loaders for image/video and warped-video datasets
   - **distributed** helpers for multi-GPU execution
   - **utils** for optical flow, LoRA, FP8, sampling, and general helpers  
   Architecturally, this package is the reusable backbone that both scripts and UI frontends import.

7. **Training subsystem: `scripts/` and `datasets/`**
   Training is split into the same two stages as inference:
   - Pass 1 trains the base VOID inpainting model
   - Pass 2 fine-tunes with warped-noise initialization  
   Dataset configuration is driven by `datasets/void_train_data.json`, which points the training scripts at generated sequences and their metadata. The training scripts are separate from data generation, so the repo keeps model optimization decoupled from synthetic dataset creation.

8. **Synthetic data generation: `data_generation/`**
   This subsystem manufactures paired “with object / without object” videos plus masks for training. It has two sources:
   - **HUMOTO + Blender**: human-object interactions with physics-based consequences when the human is removed
   - **Kubric**: object interaction scenes using Google-scanned objects and simulated motion  
   Both produce a common training format: `rgb_full.mp4`, `rgb_removed.mp4`, `mask.mp4`, and `metadata.json`. This common output contract is what feeds the training scripts.

9. **External dependencies that shape the architecture**
   A few external systems materially affect the design:
   - **Gemini API** is required for VLM-based mask reasoning in Stage 2 of mask generation
   - **SAM2** is a separate dependency for object segmentation
   - **Blender** is required for HUMOTO-based data synthesis
   - **Kubric/PyBullet** are used for physics-based object interaction generation
   - **HuggingFace checkpoints** provide the base CogVideoX inpainting model and VOID Pass 1/Pass 2 weights  
   These are not incidental; they define where the repo ends and external systems begin.

10. **Sample and config layout**
   `sample/` contains ready-made input sequences showing the expected runtime contract: `input_video.mp4`, `prompt.json`, and `quadmask_0.mp4`. `config/` holds model/runtime presets for different inference modes, including quadmask-conditioned CogVideoX and Pass 2 refinement settings. These files act as the glue between data artifacts and the model runtime.

11. **Auxiliary UI and demo surfaces**
   `videox_fun/ui/` and `videox_fun/api/` provide reusable UI/API entry points around the same core model stack. They are not the main VOID pipeline, but they show the repo supports both scripted batch execution and interactive/demo usage, including a Gradio-style experience.

12. **Overall data flow**
   The clearest end-to-end diagram is:
   raw video and prompt → human seed points → SAM2 mask → VLM interaction reasoning → quadmask → CogVideoX-based Pass 1 inpainting → optional warped-noise Pass 2 refinement → output video.  
   For training, the flow is synthetic scene generation → paired counterfactual videos + mask → dataset metadata → Pass 1/Pass 2 training → saved checkpoints used by the inference path.

Main entry points: [`inference/cogvideox_fun/inference_with_pass1_warped_noise.py`](https://github.com/netflix/void-model/blob/main/inference/cogvideox_fun/inference_with_pass1_warped_noise.py) (Pass 1), [`videox_fun/pipeline/pipeline_cogvideox_fun_inpaint.py`](https://github.com/netflix/void-model/blob/main/videox_fun/pipeline/pipeline_cogvideox_fun_inpaint.py) (Inpaint pipeline), [`VLM-MASK-REASONER/stage1_sam2_segmentation.py`](https://github.com/netflix/void-model/blob/main/VLM-MASK-REASONER/stage1_sam2_segmentation.py) (SAM2 mask), [`videox_fun/pipeline/pipeline_cogvideox_fun.py`](https://github.com/netflix/void-model/blob/main/videox_fun/pipeline/pipeline_cogvideox_fun.py) (Base pipeline)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_mask_generation["Mask generation"]
  node_mask_orchestrator["Mask pipeline<br/>preprocess pipeline"]
  node_point_gui(("Point picker<br/>annotation gui"))
  node_sam2_stage["SAM2 mask<br/>segmentation stage"]
  node_vlm_stage["VLM reasoning<br/>reasoning stage"]
  node_grey_stage["Grey masks<br/>expansion stage"]
  node_combine_stage["Quadmask combine<br/>mask composer"]
  node_quadmask_edit(("Quadmask edit<br/>correction gui<br/>[edit_quadmask.py]"))
end

subgraph group_inference["Inference"]
  node_pass1_infer["Pass 1<br/>inpaint runtime"]
  node_warp_noise["Warped noise<br/>refinement input"]
  node_pass2_refine["Pass 2<br/>refine runtime<br/>[pass_2_refine.sh]"]
end

subgraph group_training_data["Training and data"]
  node_paired_renderer["Paired render<br/>data synthesis"]
  node_kubric_scene["Kubric scenes<br/>scene generator"]
  node_train_config["Train config<br/>dataset spec"]
  node_train_pass1["Train Pass 1<br/>training script<br/>[train.py]"]
  node_train_pass2["Train Pass 2<br/>training script"]
end

subgraph group_core_runtime["Core runtime"]
  node_sample_input["Input sample<br/>sample contract<br/>[input_video.mp4]"]
  node_sample_prompt["Prompt JSON<br/>sample contract<br/>[prompt.json]"]
  node_sample_quadmask["Quadmask sample<br/>sample contract<br/>[quadmask_0.mp4]"]
  node_pipeline_inpaint["Inpaint pipeline"]
  node_base_pipeline["Base pipeline"]
  node_transformer["Transformer<br/>model"]
  node_vae["VAE<br/>model<br/>[cogvideox_vae.py]"]
  node_video_data["Video data<br/>dataset layer<br/>[dataset_video.py]"]
  node_api_layer["API layer<br/>serving<br/>[api.py]"]
  node_ui_layer["UI layer<br/>demo ui<br/>[ui.py]"]
end

subgraph group_external_deps["External dependencies"]
  node_sam2_ext(("SAM2<br/>external segmenter"))
  node_gemini_ext(("Gemini API<br/>external VLM"))
  node_blender_ext(("Blender<br/>external simulator"))
  node_kubric_ext(("Kubric/PyBullet<br/>external physics"))
  node_hf_ext(("HF checkpoints<br/>external weights"))
end

node_sample_input -->|"video"| node_pass1_infer
node_sample_prompt -->|"prompt"| node_pass1_infer
node_sample_quadmask -->|"conditioning"| node_pass1_infer
node_point_gui -->|"seed points"| node_sam2_stage
node_sam2_stage -->|"object mask"| node_vlm_stage
node_vlm_stage -->|"interaction regions"| node_grey_stage
node_grey_stage -->|"expanded mask"| node_combine_stage
node_sam2_stage -->|"object mask"| node_combine_stage
node_combine_stage -->|"quadmask mp4"| node_quadmask_edit
node_quadmask_edit -->|"edited quadmask"| node_pass1_infer
node_pass1_infer -->|"pass1 result"| node_warp_noise
node_warp_noise -->|"refinement input"| node_pass2_refine
node_pass1_infer -->|"video state"| node_pass2_refine
node_paired_renderer -->|"paired videos"| node_train_config
node_kubric_scene -->|"scenes"| node_paired_renderer
node_train_config -->|"dataset manifest"| node_train_pass1
node_train_config -->|"dataset manifest"| node_train_pass2
node_train_pass1 -->|"optimizes"| node_pipeline_inpaint
node_train_pass2 -->|"refines"| node_pipeline_inpaint
node_pipeline_inpaint -->|"builds on"| node_base_pipeline
node_base_pipeline -->|"uses"| node_transformer
node_base_pipeline -->|"uses"| node_vae
node_video_data -->|"loads"| node_train_pass1
node_video_data -->|"loads"| node_train_pass2
node_api_layer -->|"serves"| node_pipeline_inpaint
node_ui_layer -->|"drives"| node_pipeline_inpaint
node_sam2_ext -.->|"dependency"| node_sam2_stage
node_gemini_ext -.->|"dependency"| node_vlm_stage
node_blender_ext -.->|"dependency"| node_paired_renderer
node_kubric_ext -.->|"dependency"| node_kubric_scene
node_hf_ext -.->|"weights"| node_base_pipeline

click node_sample_input "https://github.com/netflix/void-model/blob/main/sample/lime/input_video.mp4"
click node_sample_prompt "https://github.com/netflix/void-model/blob/main/sample/lime/prompt.json"
click node_sample_quadmask "https://github.com/netflix/void-model/blob/main/sample/lime/quadmask_0.mp4"
click node_mask_orchestrator "https://github.com/netflix/void-model/tree/main/VLM-MASK-REASONER"
click node_point_gui "https://github.com/netflix/void-model/blob/main/VLM-MASK-REASONER/point_selector_gui.py"
click node_sam2_stage "https://github.com/netflix/void-model/blob/main/VLM-MASK-REASONER/stage1_sam2_segmentation.py"
click node_vlm_stage "https://github.com/netflix/void-model/blob/main/VLM-MASK-REASONER/stage2_vlm_analysis.py"
click node_grey_stage "https://github.com/netflix/void-model/blob/main/VLM-MASK-REASONER/stage3a_generate_grey_masks.py"
click node_combine_stage "https://github.com/netflix/void-model/blob/main/VLM-MASK-REASONER/stage4_combine_masks.py"
click node_quadmask_edit "https://github.com/netflix/void-model/blob/main/VLM-MASK-REASONER/edit_quadmask.py"
click node_pass1_infer "https://github.com/netflix/void-model/blob/main/inference/cogvideox_fun/inference_with_pass1_warped_noise.py"
click node_warp_noise "https://github.com/netflix/void-model/blob/main/inference/cogvideox_fun/make_warped_noise.py"
click node_pass2_refine "https://github.com/netflix/void-model/blob/main/inference/pass_2_refine.sh"
click node_paired_renderer "https://github.com/netflix/void-model/blob/main/data_generation/render_paired_videos_blender_quadmask.py"
click node_kubric_scene "https://github.com/netflix/void-model/blob/main/data_generation/kubric_variable_objects.py"
click node_train_config "https://github.com/netflix/void-model/blob/main/datasets/void_train_data.json"
click node_train_pass1 "https://github.com/netflix/void-model/blob/main/scripts/cogvideox_fun/train.py"
click node_train_pass2 "https://github.com/netflix/void-model/blob/main/scripts/cogvideox_fun/train_warped_noise.py"
click node_pipeline_inpaint "https://github.com/netflix/void-model/blob/main/videox_fun/pipeline/pipeline_cogvideox_fun_inpaint.py"
click node_base_pipeline "https://github.com/netflix/void-model/blob/main/videox_fun/pipeline/pipeline_cogvideox_fun.py"
click node_transformer "https://github.com/netflix/void-model/blob/main/videox_fun/models/cogvideox_transformer3d.py"
click node_vae "https://github.com/netflix/void-model/blob/main/videox_fun/models/cogvideox_vae.py"
click node_video_data "https://github.com/netflix/void-model/blob/main/videox_fun/data/dataset_video.py"
click node_api_layer "https://github.com/netflix/void-model/blob/main/videox_fun/api/api.py"
click node_ui_layer "https://github.com/netflix/void-model/blob/main/videox_fun/ui/ui.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_mask_orchestrator,node_point_gui,node_sam2_stage,node_vlm_stage,node_grey_stage,node_combine_stage,node_quadmask_edit toneBlue
class node_pass1_infer,node_warp_noise,node_pass2_refine toneAmber
class node_paired_renderer,node_kubric_scene,node_train_config,node_train_pass1,node_train_pass2 toneMint
class node_sample_input,node_sample_prompt,node_sample_quadmask,node_pipeline_inpaint,node_base_pipeline,node_transformer,node_vae,node_video_data,node_api_layer,node_ui_layer toneRose
class node_sam2_ext,node_gemini_ext,node_blender_ext,node_kubric_ext,node_hf_ext toneIndigo
```

## Components

### Mask generation

Human-in-the-loop pipeline that turns raw video into quadmask conditioning.

- [Mask pipeline](https://github.com/netflix/void-model/tree/main/VLM-MASK-REASONER): preprocess pipeline · End-to-end mask generation flow from seed points to quadmask video. · `VLM-MASK-REASONER`
- [Point picker](https://github.com/netflix/void-model/blob/main/VLM-MASK-REASONER/point_selector_gui.py): annotation gui · Human seed-point selection for initial object localization. · `VLM-MASK-REASONER/point_selector_gui.py`
- [SAM2 mask](https://github.com/netflix/void-model/blob/main/VLM-MASK-REASONER/stage1_sam2_segmentation.py): segmentation stage · Produces the object mask from selected points. · `VLM-MASK-REASONER/stage1_sam2_segmentation.py`
- [VLM reasoning](https://github.com/netflix/void-model/blob/main/VLM-MASK-REASONER/stage2_vlm_analysis.py): reasoning stage · Uses a vision-language model to infer physically affected regions. · `VLM-MASK-REASONER/stage2_vlm_analysis.py`
- [Grey masks](https://github.com/netflix/void-model/blob/main/VLM-MASK-REASONER/stage3a_generate_grey_masks.py): expansion stage · Expands interaction regions into grey-mask conditioning. · `VLM-MASK-REASONER/stage3a_generate_grey_masks.py`
- [Quadmask combine](https://github.com/netflix/void-model/blob/main/VLM-MASK-REASONER/stage4_combine_masks.py): mask composer · Combines object, preserved, and interaction regions into quadmask output. · `VLM-MASK-REASONER/stage4_combine_masks.py`
- [Quadmask edit](https://github.com/netflix/void-model/blob/main/VLM-MASK-REASONER/edit_quadmask.py): correction gui · Manual correction tool for generated quadmasks. · `VLM-MASK-REASONER/edit_quadmask.py`

### Inference

Two-pass video inpainting runtime for VOID outputs.

- [Pass 1](https://github.com/netflix/void-model/blob/main/inference/cogvideox_fun/inference_with_pass1_warped_noise.py): inpaint runtime · Primary CogVideoX-Fun inpainting inference conditioned on quadmask and prompt. · `inference/cogvideox_fun/inference_with_pass1_warped_noise.py`
- [Warped noise](https://github.com/netflix/void-model/blob/main/inference/cogvideox_fun/make_warped_noise.py): refinement input · Noise derived from Pass 1 to stabilize the second pass. · `inference/cogvideox_fun/make_warped_noise.py`
- [Pass 2](https://github.com/netflix/void-model/blob/main/inference/pass_2_refine.sh): refine runtime · Optional refinement step for temporal consistency on long clips. · `inference/pass_2_refine.sh`

### Training and data

Synthetic paired-video generation plus training entry points and dataset metadata.

- [Paired render](https://github.com/netflix/void-model/blob/main/data_generation/render_paired_videos_blender_quadmask.py): data synthesis · Produces with-object and without-object videos plus masks for training. · `data_generation/render_paired_videos_blender_quadmask.py`
- [Kubric scenes](https://github.com/netflix/void-model/blob/main/data_generation/kubric_variable_objects.py): scene generator · Physics-driven object interaction scene generation. · `data_generation/kubric_variable_objects.py`
- [Train config](https://github.com/netflix/void-model/blob/main/datasets/void_train_data.json): dataset spec · Dataset manifest that maps training scripts to generated artifacts. · `datasets/void_train_data.json`
- [Train Pass 1](https://github.com/netflix/void-model/blob/main/scripts/cogvideox_fun/train.py): training script · Optimizes the base VOID inpainting model. · `scripts/cogvideox_fun/train.py`
- [Train Pass 2](https://github.com/netflix/void-model/blob/main/scripts/cogvideox_fun/train_warped_noise.py): training script · Fine-tunes the model with warped-noise initialization. · `scripts/cogvideox_fun/train_warped_noise.py`

### Core runtime

Shared model, data, UI, and API layers used by training and inference.

- [Input sample](https://github.com/netflix/void-model/blob/main/sample/lime/input_video.mp4): sample contract · Example runtime input video and prompt pair. · `sample/lime/input_video.mp4`
- [Prompt JSON](https://github.com/netflix/void-model/blob/main/sample/lime/prompt.json): sample contract · Example text prompt metadata for inference. · `sample/lime/prompt.json`
- [Quadmask sample](https://github.com/netflix/void-model/blob/main/sample/lime/quadmask_0.mp4): sample contract · Example quadmask conditioning artifact expected by inference. · `sample/lime/quadmask_0.mp4`
- [Inpaint pipeline](https://github.com/netflix/void-model/blob/main/videox_fun/pipeline/pipeline_cogvideox_fun_inpaint.py): pipeline · Shared inpainting pipeline used by VOID inference and training. · `videox_fun/pipeline/pipeline_cogvideox_fun_inpaint.py`
- [Base pipeline](https://github.com/netflix/void-model/blob/main/videox_fun/pipeline/pipeline_cogvideox_fun.py): pipeline · Underlying CogVideoX-Fun generation pipeline. · `videox_fun/pipeline/pipeline_cogvideox_fun.py`
- [Transformer](https://github.com/netflix/void-model/blob/main/videox_fun/models/cogvideox_transformer3d.py): model · Core 3D transformer backbone for video synthesis. · `videox_fun/models/cogvideox_transformer3d.py`
- [VAE](https://github.com/netflix/void-model/blob/main/videox_fun/models/cogvideox_vae.py): model · Latent video encoder/decoder used by the runtime. · `videox_fun/models/cogvideox_vae.py`
- [Video data](https://github.com/netflix/void-model/blob/main/videox_fun/data/dataset_video.py): dataset layer · Dataset loaders for video and warped-video training/inference. · `videox_fun/data/dataset_video.py`
- [API layer](https://github.com/netflix/void-model/blob/main/videox_fun/api/api.py): serving · Programmatic interface over the shared model stack. · `videox_fun/api/api.py`
- [UI layer](https://github.com/netflix/void-model/blob/main/videox_fun/ui/ui.py): demo ui · Interactive frontend for model usage and demos. · `videox_fun/ui/ui.py`

### External dependencies

Required third-party systems that define major architectural boundaries.

- **SAM2**: external segmenter · Third-party object segmentation dependency.
- **Gemini API**: external VLM · Vision-language reasoning service used for interaction analysis.
- **Blender**: external simulator · Required renderer and scene engine for HUMOTO-style synthesis.
- **Kubric/PyBullet**: external physics · Physics and scene-generation stack for synthetic data.
- **HF checkpoints**: external weights · Base and adapted model weights sourced from HuggingFace.

## Connections

- Input sample → Pass 1: video · Inference consumes the input clip.
- Prompt JSON → Pass 1: prompt · Text conditioning for inpainting.
- Quadmask sample → Pass 1: conditioning · Quadmask guides object removal and preservation.
- Point picker → SAM2 mask: seed points · Human annotation initializes object segmentation.
- SAM2 mask → VLM reasoning: object mask · Segmented object region is passed to VLM reasoning.
- VLM reasoning → Grey masks: interaction regions · Reasoned physical influence areas are expanded into grey masks.
- Grey masks → Quadmask combine: expanded mask · Grey-mask outputs feed quadmask composition.
- SAM2 mask → Quadmask combine: object mask · Object region is one quadmask input.
- Quadmask combine → Quadmask edit: quadmask mp4 · Generated quadmask can be manually corrected.
- Quadmask edit → Pass 1: edited quadmask · Corrected conditioning is used by inference.
- Pass 1 → Warped noise: pass1 result · Pass 1 output is transformed into warped noise.
- Warped noise → Pass 2: refinement input · Pass 2 uses warped-noise initialization.
- Pass 1 → Pass 2: video state · Second pass refines the first-pass result.
- Paired render → Train config: paired videos · Synthetic outputs are registered for training.
- Kubric scenes → Paired render: scenes · Physics scenes feed paired-video rendering.
- Train config → Train Pass 1: dataset manifest · Pass 1 training reads the dataset configuration.
- Train config → Train Pass 2: dataset manifest · Pass 2 fine-tuning uses the same dataset contract.
- Train Pass 1 → Inpaint pipeline: optimizes · Training updates the inpainting pipeline weights.
- Train Pass 2 → Inpaint pipeline: refines · Warped-noise training improves the same runtime.
- Inpaint pipeline → Base pipeline: builds on · Inpainting pipeline wraps the base generation stack.
- Base pipeline → Transformer: uses · Transformer backbone drives generation.
- Base pipeline → VAE: uses · Latent encoding/decoding is shared by the pipelines.
- Video data → Train Pass 1: loads · Training consumes standard video datasets.
- Video data → Train Pass 2: loads · Warped-video datasets support refinement training.
- API layer → Inpaint pipeline: serves · API exposes the core inpainting stack.
- UI layer → Inpaint pipeline: drives · Interactive UI invokes the same runtime.
- SAM2 → SAM2 mask: dependency · External segmenter is required by stage 1.
- Gemini API → VLM reasoning: dependency · VLM reasoning depends on Gemini.
- Blender → Paired render: dependency · Blender is required for synthetic rendering.
- Kubric/PyBullet → Kubric scenes: dependency · Kubric/PyBullet drive scene physics.
- HF checkpoints → Base pipeline: weights · Base model checkpoints are sourced externally.
