How irom-princeton/c-cubed works
1. Repository purpose
- This is a Python package, dynuq, for calibrated controllable video generation.
- The core idea is to produce videos plus dense uncertainty / confidence predictions in latent space, then project those uncertainties into interpretable RGB-space heatmaps.
2. Top-level execution modes
- The repo has three main runtime paths:
- Training a video model with calibration-aware losses.
- Inference to generate videos and dense confidence maps.
- Metric computation to evaluate calibration quality after generation.
- These are exposed through scripts under dynuq/scripts and wrapped by shell scripts for local or Slurm execution.
3. Data ingestion boundary
- dynuq/datasets/openxmp4videodataset.py is the dataset entry point.
- It likely handles video-file loading and prepares the conditioning inputs needed by the model, such as control signals or actions, as implied by the repo’s controllable video-generation focus.
- This is the main boundary between raw dataset storage and model-ready tensors.
4. Representation / tokenization layer
- dynuq/tokenizers/vae.py and dynuq/tokenizers/cosmos_tokenizer.py define the latent-space interface.
- The architecture is centered on operating in latent space rather than pixel space, which the README calls out as important for stability and cost.
- This layer converts video frames into compact representations and back, and is also the bridge for turning latent uncertainty into visualizations.
5. Generative model layer
- dynuq/models/model.py and dynuq/models/diffusion.py contain the main generative model implementation.
- The project uses a diffusion-style video generator as the core predictor.
- This is the subsystem that consumes latent tokens and conditioning inputs and produces future video latents, plus uncertainty-related outputs.
6. Calibration-aware training logic
- dynuq/loss_wrappers/ contains the training objective abstractions:
- default_loss_wrapper.py
- calibration_loss_wrapper.py
- base_loss_wrapper.py
- This layer sits between the model and the optimization loop.
- Its role is to combine prediction loss with calibration/correctness objectives, reflecting the README’s emphasis on strictly proper scoring rules.
7. Training orchestration
- dynuq/scripts/train.py is the main training entry point.
- dynuq/scripts/utils/train_utils.py contains support code for setup, loading, and training-time utilities.
- The training flow is: config → dataset → tokenizer/model → loss wrapper → optimizer loop → checkpoints/artifacts.
- A sample config is provided in dynuq/configs/train_bridge_256_example.yaml.
8. Inference / generation path
- dynuq/scripts/example_dataset_inference.py runs example inference over a dataset.
- The corresponding configs in dynuq/configs/bridge_inference_example.yaml drive this mode.
- This path produces generated videos together with dense confidence predictions, which are likely used both for visualization and downstream evaluation.
9. Evaluation and metrics path
- dynuq/scripts/compute_metrics.py computes uncertainty and calibration metrics after inference.
- dynuq/metrics/base_metrics.py and dynuq/metrics/calibration_metrics.py define reusable metric logic.
- dynuq/evaluators/base_evaluator.py and dynuq/evaluators/bridge_evaluator.py provide dataset/task-specific evaluation orchestration.
- The metrics config tree under dynuq/configs/metrics/ and dynuq/configs/evaluators/ separates generic metric definitions from task-specific evaluation wiring.
10. Calibration-specific subsystem
- The repo has a distinct calibration-aware branch of logic, not just generic generation.
- dynuq/loss_wrappers/calibration_loss_wrapper.py handles training objectives for calibration.
- dynuq/metrics/calibration_metrics.py handles posthoc measurement of calibration quality.
- The README’s described output is a dense uncertainty estimate, so this subsystem is central rather than auxiliary.
11. Configuration architecture
- YAML configs are a major integration boundary:
- training configs in dynuq/configs/
- evaluator configs in dynuq/configs/evaluators/
- metric configs in dynuq/configs/metrics/
- The repo appears designed so experiments can be swapped by changing config files rather than code.
- This makes the architecture highly experiment-driven and reproducible.
12. Operational wrappers
- bash_scripts/ contains local test wrappers for inference and metric computation.
- slurm_scripts/ contains batch-job equivalents for cluster execution, including bridge_train_8gpu.sh.
- The README indicates the default operational environment is a Slurm cluster, with the option to run the same scripts locally by replacing sbatch with bash.
13. External/runtime dependencies that matter architecturally
- The project is packaged as an installable Python distribution via pyproject.toml.
- The README specifies Python 3.13 and uv for environment management.
- The runtime assumes access to substantial compute, especially for training and evaluation on Bridge-style robot video datasets.
14. End-to-end data flow
- Raw video dataset → dataset loader → latent tokenizer/VAE → diffusion video model → calibration-aware loss during training.
- At inference, the same model produces video outputs plus dense uncertainty estimates.
- A separate evaluation path consumes those outputs and computes calibration metrics, closing the loop between generation quality and uncertainty quality.
Main entry points: train.py Train entry · compute_metrics.py Metrics entry · model.py Video model · example_dataset_inference.py Inference entry
Components 26
Runtime paths
Main execution entry points for training, inference, and evaluation.
- Train entry · Main training entry point for the calibrated video model.
dynuq/scripts/train.py - Inference entry · Runs example dataset inference and emits generated videos plus confidence maps.
dynuq/scripts/example_dataset_inference.py - Train utils · Utilities for training setup, loading, and orchestration.
dynuq/scripts/utils/train_utils.py
Core model stack
Data loading, tokenization, model, and training objectives.
- Video data · Loads video data and prepares model inputs from the dataset boundary.
dynuq/datasets/openxmp4videodataset.py - VAE tokenizer · Encodes and decodes video frames in latent space.
dynuq/tokenizers/vae.py - Cosmos tokenizer · Alternative latent tokenization interface for video representations.
dynuq/tokenizers/cosmos_tokenizer.py - Video model · Main controllable video generation model and prediction interface.
dynuq/models/model.py - Diffusion core · Diffusion-style generative backbone used by the video model.
dynuq/models/diffusion.py - Base loss · Shared loss-wrapper interface for training objectives.
dynuq/loss_wrappers/base_loss_wrapper.py - Default loss · Standard prediction loss wrapper used in training.
dynuq/loss_wrappers/default_loss_wrapper.py - Calib loss · Calibration-aware objective combining prediction and uncertainty quality.
dynuq/loss_wrappers/calibration_loss_wrapper.py
Evaluation stack
Metric computation and evaluator orchestration.
- Metrics entry · Computes calibration and uncertainty metrics after inference.
dynuq/scripts/compute_metrics.py - Base metrics · Shared metric utilities and interfaces.
dynuq/metrics/base_metrics.py - Calib metrics · Posthoc calibration and uncertainty metric computations.
dynuq/metrics/calibration_metrics.py - Base eval · Shared evaluator interface for task-specific evaluation.
dynuq/evaluators/base_evaluator.py - Bridge eval · Bridge-specific evaluation wiring for the dataset/task.
dynuq/evaluators/bridge_evaluator.py
Operational wrappers
Shell wrappers for local and Slurm execution.
- Local infer · Local wrapper for running example inference.
bash_scripts/test/bridge_inference_example.sh - Local metrics · Local wrapper for running metric computation.
bash_scripts/test/bridge_compute_metrics_example.sh - Slurm train · Cluster training wrapper for multi-GPU execution.
slurm_scripts/train/bridge_train_8gpu.sh - Slurm infer · Cluster wrapper for inference runs.
slurm_scripts/test/bridge_inference.sh - Slurm metrics · Cluster wrapper for metric computation runs.
slurm_scripts/test/bridge_compute_metrics.sh
Config files
Experiment and evaluation configuration boundaries.
- Train config · Example training configuration for Bridge 256 setup.
dynuq/configs/train_bridge_256_example.yaml - Infer config · Example inference configuration for dataset generation.
dynuq/configs/bridge_inference_example.yaml - Metrics config · Example configuration for metric computation.
dynuq/configs/bridge_compute_metrics_example.yaml - Eval config · Task-specific evaluator configuration.
dynuq/configs/evaluators/bridge_evaluator.yaml - Calib cfg · Calibration metric configuration definition.
dynuq/configs/metrics/calibration_metrics/calibration_metrics.yaml
Connections 29
- Video data → VAE tokenizer: video tensors · Raw video data is converted into latent representations.
- Video data → Cosmos tokenizer: alt latent path · Dataset inputs can also feed the alternative tokenizer interface.
- VAE tokenizer → Video model: latents · Latent tokens are consumed by the generative model.
- Cosmos tokenizer → Video model: tokens · Alternative latent tokens feed the main model.
- Diffusion core → Video model: backbone · Diffusion implementation underpins the main model.
- Video model → Base loss: predictions · Model outputs are scored through the shared loss abstraction.
- Base loss → Default loss: default · Default training objective specialization.
- Base loss → Calib loss: calibration · Calibration-aware objective specialization.
- Train config → Train entry: train config · Training execution is driven by the example config.
- Train entry → Train utils: setup · Training entry point uses shared training utilities.
- Train entry → Video data: load data · Training pulls data from the dataset boundary.
- Train entry → Video model: fit model · Training optimizes the generative model.
- Train entry → Calib loss: train loss · Training can use calibration-aware objectives.
- Infer config → Inference entry: infer config · Inference execution is driven by the example config.
- Inference entry → Video model: generate · Inference runs the model to produce video and uncertainty outputs.
- Inference entry → VAE tokenizer: decode · Generated latents are projected back for visualization.
- Metrics config → Metrics entry: metrics config · Metric computation is driven by the example config.
- Metrics entry → Base metrics: metric API · Metric entry point uses the shared metric abstraction.
- Metrics entry → Calib metrics: calib metrics · Calibration-specific metrics are computed here.
- Metrics entry → Bridge eval: task eval · Metric computation is coordinated with the Bridge evaluator.
- Bridge eval → Base eval: implements · Bridge evaluator specializes the base evaluator interface.
- Eval config → Bridge eval: eval wiring · Evaluator configuration selects task-specific evaluation behavior.
- Calib cfg → Calib metrics: metric wiring · Calibration metric configuration selects the metric suite.
- Local infer → Inference entry: local run · Local shell wrapper launches inference.
- Local metrics → Metrics entry: local run · Local shell wrapper launches metric computation.
- Slurm train → Train entry: sbatch · Slurm job wrapper launches training.
- Slurm infer → Inference entry: sbatch · Slurm job wrapper launches inference.
- Slurm metrics → Metrics entry: sbatch · Slurm job wrapper launches metric computation.
- Calib loss → Calib metrics: aligned with · Training objectives and posthoc metrics reflect the same calibration focus.