How netflix/void-model works
1. What this repo is
VOID is a video object-removal system with two distinct halves: a mask-generation pipeline that finds the object and its interaction regions, and a video inpainting pipeline that synthesizes the cleaned video. The central architectural idea is the quadmask: a 4-valued per-pixel conditioning signal that tells the model what to delete, what to preserve, and what interaction area to reconstruct.
2. Top-level pipeline boundary
The repository is organized around three flows:
- Mask creation: raw video → point selection → SAM2 segmentation → VLM reasoning → quadmask
- Inference: video + quadmask + background prompt → Pass 1 inpainting → optional Pass 2 refinement
- Training/data generation: synthetic paired videos + masks → training metadata → Pass 1/Pass 2 model training
This separation is important for architecture diagrams because mask generation, model inference, and synthetic data production are all independent subsystems.
3. Mask-generation subsystem: VLM-MASK-REASONER/
This directory implements a four-stage preprocessing pipeline. A human first selects seed points in a GUI, then SAM2 produces the object mask, a Gemini/VLM stage reasons about physically affected regions, a grey-mask stage expands those regions, and a final combiner produces the quadmask video. The output artifact is quadmask_0.mp4, which becomes the primary conditioning input for inference.
4. Interactive tools for mask refinement
Two GUIs sit on the mask boundary:
- point_selector_gui.py for initial point annotation
- edit_quadmask.py for manual correction of the generated quadmask
These are human-in-the-loop correction points, not part of the learned model, and they exist to improve the quality of the conditioning signal before inference.
5. Inference subsystem: inference/
Inference is explicitly two-pass:
- Pass 1 uses the base CogVideoX-Fun inpainting model adapted for VOID, conditioned on the quadmask and text prompt.
- Pass 2 refines the result using warped noise derived from Pass 1, improving temporal consistency over long clips.
This two-pass design is the main runtime architecture for output generation.
6. Core model package: videox_fun/
This is the shared model/runtime layer used by training, inference, and UI. It contains:
- pipelines for CogVideoX-Fun and inpainting
- models for the transformer, VAE, and cache utilities
- data loaders for image/video and warped-video datasets
- distributed helpers for multi-GPU execution
- utils for optical flow, LoRA, FP8, sampling, and general helpers
Architecturally, this package is the reusable backbone that both scripts and UI frontends import.
7. Training subsystem: scripts/ and datasets/
Training is split into the same two stages as inference:
- Pass 1 trains the base VOID inpainting model
- Pass 2 fine-tunes with warped-noise initialization
Dataset configuration is driven by datasets/void_train_data.json, which points the training scripts at generated sequences and their metadata. The training scripts are separate from data generation, so the repo keeps model optimization decoupled from synthetic dataset creation.
8. Synthetic data generation: data_generation/
This subsystem manufactures paired “with object / without object” videos plus masks for training. It has two sources:
- HUMOTO + Blender: human-object interactions with physics-based consequences when the human is removed
- Kubric: object interaction scenes using Google-scanned objects and simulated motion
Both produce a common training format: rgb_full.mp4, rgb_removed.mp4, mask.mp4, and metadata.json. This common output contract is what feeds the training scripts.
9. External dependencies that shape the architecture
A few external systems materially affect the design:
- Gemini API is required for VLM-based mask reasoning in Stage 2 of mask generation
- SAM2 is a separate dependency for object segmentation
- Blender is required for HUMOTO-based data synthesis
- Kubric/PyBullet are used for physics-based object interaction generation
- HuggingFace checkpoints provide the base CogVideoX inpainting model and VOID Pass 1/Pass 2 weights
These are not incidental; they define where the repo ends and external systems begin.
10. Sample and config layout
sample/ contains ready-made input sequences showing the expected runtime contract: input_video.mp4, prompt.json, and quadmask_0.mp4. config/ holds model/runtime presets for different inference modes, including quadmask-conditioned CogVideoX and Pass 2 refinement settings. These files act as the glue between data artifacts and the model runtime.
11. Auxiliary UI and demo surfaces
videox_fun/ui/ and videox_fun/api/ provide reusable UI/API entry points around the same core model stack. They are not the main VOID pipeline, but they show the repo supports both scripted batch execution and interactive/demo usage, including a Gradio-style experience.
12. Overall data flow
The clearest end-to-end diagram is:
raw video and prompt → human seed points → SAM2 mask → VLM interaction reasoning → quadmask → CogVideoX-based Pass 1 inpainting → optional warped-noise Pass 2 refinement → output video.
For training, the flow is synthetic scene generation → paired counterfactual videos + mask → dataset metadata → Pass 1/Pass 2 training → saved checkpoints used by the inference path.
Main entry points: inference_with_pass1_warped_noise.py Pass 1 · pipeline_cogvideox_fun_inpaint.py Inpaint pipeline · stage1_sam2_segmentation.py SAM2 mask · pipeline_cogvideox_fun.py Base pipeline
Components 30
Mask generation
Human-in-the-loop pipeline that turns raw video into quadmask conditioning.
- Mask pipeline · End-to-end mask generation flow from seed points to quadmask video.
VLM-MASK-REASONER - Point picker · Human seed-point selection for initial object localization.
VLM-MASK-REASONER/point_selector_gui.py - SAM2 mask · Produces the object mask from selected points.
VLM-MASK-REASONER/stage1_sam2_segmentation.py - VLM reasoning · Uses a vision-language model to infer physically affected regions.
VLM-MASK-REASONER/stage2_vlm_analysis.py - Grey masks · Expands interaction regions into grey-mask conditioning.
VLM-MASK-REASONER/stage3a_generate_grey_masks.py - Quadmask combine · Combines object, preserved, and interaction regions into quadmask output.
VLM-MASK-REASONER/stage4_combine_masks.py - Quadmask edit · Manual correction tool for generated quadmasks.
VLM-MASK-REASONER/edit_quadmask.py
Inference
Two-pass video inpainting runtime for VOID outputs.
- Pass 1 · Primary CogVideoX-Fun inpainting inference conditioned on quadmask and prompt.
inference/cogvideox_fun/inference_with_pass1_warped_noise.py - Warped noise · Noise derived from Pass 1 to stabilize the second pass.
inference/cogvideox_fun/make_warped_noise.py - Pass 2 · Optional refinement step for temporal consistency on long clips.
inference/pass_2_refine.sh
Training and data
Synthetic paired-video generation plus training entry points and dataset metadata.
- Paired render · Produces with-object and without-object videos plus masks for training.
data_generation/render_paired_videos_blender_quadmask.py - Kubric scenes · Physics-driven object interaction scene generation.
data_generation/kubric_variable_objects.py - Train config · Dataset manifest that maps training scripts to generated artifacts.
datasets/void_train_data.json - Train Pass 1 · Optimizes the base VOID inpainting model.
scripts/cogvideox_fun/train.py - Train Pass 2 · Fine-tunes the model with warped-noise initialization.
scripts/cogvideox_fun/train_warped_noise.py
Core runtime
Shared model, data, UI, and API layers used by training and inference.
- Input sample · Example runtime input video and prompt pair.
sample/lime/input_video.mp4 - Prompt JSON · Example text prompt metadata for inference.
sample/lime/prompt.json - Quadmask sample · Example quadmask conditioning artifact expected by inference.
sample/lime/quadmask_0.mp4 - Inpaint pipeline · Shared inpainting pipeline used by VOID inference and training.
videox_fun/pipeline/pipeline_cogvideox_fun_inpaint.py - Base pipeline · Underlying CogVideoX-Fun generation pipeline.
videox_fun/pipeline/pipeline_cogvideox_fun.py - Transformer · Core 3D transformer backbone for video synthesis.
videox_fun/models/cogvideox_transformer3d.py - VAE · Latent video encoder/decoder used by the runtime.
videox_fun/models/cogvideox_vae.py - Video data · Dataset loaders for video and warped-video training/inference.
videox_fun/data/dataset_video.py - API layer · Programmatic interface over the shared model stack.
videox_fun/api/api.py - UI layer · Interactive frontend for model usage and demos.
videox_fun/ui/ui.py
External dependencies
Required third-party systems that define major architectural boundaries.
- SAM2 · Third-party object segmentation dependency.
- Gemini API · Vision-language reasoning service used for interaction analysis.
- Blender · Required renderer and scene engine for HUMOTO-style synthesis.
- Kubric/PyBullet · Physics and scene-generation stack for synthetic data.
- HF checkpoints · Base and adapted model weights sourced from HuggingFace.
Connections 31
- Input sample → Pass 1: video · Inference consumes the input clip.
- Prompt JSON → Pass 1: prompt · Text conditioning for inpainting.
- Quadmask sample → Pass 1: conditioning · Quadmask guides object removal and preservation.
- Point picker → SAM2 mask: seed points · Human annotation initializes object segmentation.
- SAM2 mask → VLM reasoning: object mask · Segmented object region is passed to VLM reasoning.
- VLM reasoning → Grey masks: interaction regions · Reasoned physical influence areas are expanded into grey masks.
- Grey masks → Quadmask combine: expanded mask · Grey-mask outputs feed quadmask composition.
- SAM2 mask → Quadmask combine: object mask · Object region is one quadmask input.
- Quadmask combine → Quadmask edit: quadmask mp4 · Generated quadmask can be manually corrected.
- Quadmask edit → Pass 1: edited quadmask · Corrected conditioning is used by inference.
- Pass 1 → Warped noise: pass1 result · Pass 1 output is transformed into warped noise.
- Warped noise → Pass 2: refinement input · Pass 2 uses warped-noise initialization.
- Pass 1 → Pass 2: video state · Second pass refines the first-pass result.
- Paired render → Train config: paired videos · Synthetic outputs are registered for training.
- Kubric scenes → Paired render: scenes · Physics scenes feed paired-video rendering.
- Train config → Train Pass 1: dataset manifest · Pass 1 training reads the dataset configuration.
- Train config → Train Pass 2: dataset manifest · Pass 2 fine-tuning uses the same dataset contract.
- Train Pass 1 → Inpaint pipeline: optimizes · Training updates the inpainting pipeline weights.
- Train Pass 2 → Inpaint pipeline: refines · Warped-noise training improves the same runtime.
- Inpaint pipeline → Base pipeline: builds on · Inpainting pipeline wraps the base generation stack.
- Base pipeline → Transformer: uses · Transformer backbone drives generation.
- Base pipeline → VAE: uses · Latent encoding/decoding is shared by the pipelines.
- Video data → Train Pass 1: loads · Training consumes standard video datasets.
- Video data → Train Pass 2: loads · Warped-video datasets support refinement training.
- API layer → Inpaint pipeline: serves · API exposes the core inpainting stack.
- UI layer → Inpaint pipeline: drives · Interactive UI invokes the same runtime.
- SAM2 → SAM2 mask: dependency · External segmenter is required by stage 1.
- Gemini API → VLM reasoning: dependency · VLM reasoning depends on Gemini.
- Blender → Paired render: dependency · Blender is required for synthetic rendering.
- Kubric/PyBullet → Kubric scenes: dependency · Kubric/PyBullet drive scene physics.
- HF checkpoints → Base pipeline: weights · Base model checkpoints are sourced externally.