How skyworkai/matrix-game works
Matrix-Game is a three-generation family of interactive world models that generates video conditioned on images/prompts and player controls. The graph follows the sampled Matrix-Game-2 image-conditioned and streaming paths, then the Matrix-Game-3 generation and asynchronous VAE decode path; Matrix-Game-1 inference and its scoring capabilities remain represented as a distinct branch. Several relationships and capabilities come from README or SOURCE INDEX references rather than complete excerpts, so uncertain links are omitted.
Main entry points: causal_inference.py Causal generation pipeline · inference_bench.py Image-to-video inference · parallel_infer.py Parallel inference · inference_interactive_pipeline.py Interactive streaming pipeline
Components 31
Matrix-Game 1
- Image-to-video inference
Matrix-Game-1/inference_bench.py - Text and image encoders
Matrix-Game-1/matrixgame/encoder_variants - Diffusion transformer
Matrix-Game-1/matrixgame/model_variants/matrixgame_dit_src/models_i2v.py - Video VAE
Matrix-Game-1/matrixgame/vae_variants/matrixgame_vae.py - Parallel inference
Matrix-Game-1/parallel_infer.py - Video visualization
Matrix-Game-1/tools/visualize.py
Matrix-Game 2
- Interactive inference
Matrix-Game-2/inference.py - Streaming inference
Matrix-Game-2/inference_streaming.py - Causal generation pipeline
Matrix-Game-2/pipeline/causal_inference.py - Model and scheduler wrapper
Matrix-Game-2/utils/wan_wrapper.py - Video diffusion model
Matrix-Game-2/wan/modules/model.py - Causal video model
Matrix-Game-2/wan/modules/causal_model.py - Action conditioning
Matrix-Game-2/wan/modules/action_module.py - Wan video VAE
Matrix-Game-2/wan/modules/vae.py - Streaming VAE decode
Matrix-Game-2/demo_utils/vae_block3.py - Control overlays
Matrix-Game-2/utils/visualize.py - Flow-match solver
Matrix-Game-2/wan/utils/fm_solvers_unipc.py
Matrix-Game 3
- Generation entrypoint
Matrix-Game-3/generate.py - Interactive streaming pipeline
Matrix-Game-3/pipeline/inference_interactive_pipeline.py - Standard generation pipeline
Matrix-Game-3/pipeline/inference_pipeline.py - Video diffusion model
Matrix-Game-3/wan/modules/model.py - VAE implementation
Matrix-Game-3/wan/modules/vae2_2.py - VAE worker config
Matrix-Game-3/pipeline/vae_config.py - Asynchronous VAE worker
Matrix-Game-3/pipeline/vae_worker.py - Sequence parallelism
Matrix-Game-3/wan/distributed/sequence_parallel.py
Evaluation
- WorldScore evaluation
Matrix-Game-1/GameWorldScore/GameWorld/__init__.py - Scene evaluation CLI
Matrix-Game-1/GameWorldScore/evaluate_per_scene.py - World quality metrics
Matrix-Game-1/GameWorldScore/GameWorld
Other components
- Player / operator
- Generated video
- Latent queue
Connections 39
- Player / operator → Image-to-video inference: provides inputs
- Image-to-video inference → Text and image encoders: loads encoders
Matrix-Game-1/inference_bench.py - Image-to-video inference → Diffusion transformer: loads transformer
Matrix-Game-1/inference_bench.py - Image-to-video inference → Video VAE: loads VAE
Matrix-Game-1/inference_bench.py - Image-to-video inference → Video visualization: renders video
Matrix-Game-1/inference_bench.py - Image-to-video inference → Generated video: generates
- Parallel inference → Text and image encoders: loads encoders
Matrix-Game-1/parallel_infer.py - Parallel inference → Video VAE: loads VAE
Matrix-Game-1/parallel_infer.py - Parallel inference → Diffusion transformer: loads transformer
Matrix-Game-1/parallel_infer.py - Parallel inference → Video visualization: renders video
Matrix-Game-1/parallel_infer.py - Player / operator → Interactive inference: sets conditions
- Player / operator → Streaming inference: sets conditions
- Interactive inference → Causal generation pipeline: runs pipeline
Matrix-Game-2/inference.py - Streaming inference → Causal generation pipeline: runs pipeline
Matrix-Game-2/inference_streaming.py - Causal generation pipeline → Model and scheduler wrapper: constructs generator
Matrix-Game-2/pipeline/causal_inference.py - Model and scheduler wrapper → Video diffusion model: uses model
Matrix-Game-2/utils/wan_wrapper.py - Model and scheduler wrapper → Causal video model: uses causal model
Matrix-Game-2/utils/wan_wrapper.py - Video diffusion model → Action conditioning: applies actions
Matrix-Game-2/wan/modules/model.py - Causal generation pipeline → Flow-match solver: denoises with
- Causal generation pipeline → Streaming VAE decode: decodes frames
- Causal generation pipeline → Control overlays: renders controls
Matrix-Game-2/pipeline/causal_inference.py - Causal generation pipeline → Generated video: produces video
- Generation entrypoint → Interactive streaming pipeline: selects pipeline
Matrix-Game-3/generate.py - Generation entrypoint → Standard generation pipeline: selects pipeline
Matrix-Game-3/generate.py - Player / operator → Generation entrypoint: provides prompts / controls
- Interactive streaming pipeline → Video diffusion model: runs denoising
Matrix-Game-3/pipeline/inference_interactive_pipeline.py - Standard generation pipeline → Video diffusion model: runs denoising
Matrix-Game-3/pipeline/inference_pipeline.py - Interactive streaming pipeline → Sequence parallelism: uses parallelism
Matrix-Game-3/pipeline/inference_interactive_pipeline.py - Standard generation pipeline → Sequence parallelism: uses parallelism
Matrix-Game-3/pipeline/inference_pipeline.py - Interactive streaming pipeline → Latent queue: enqueues latents
Matrix-Game-3/pipeline/inference_interactive_pipeline.py - Standard generation pipeline → Latent queue: enqueues latents
Matrix-Game-3/pipeline/inference_pipeline.py - Latent queue → Asynchronous VAE worker: delivers latents
Matrix-Game-3/pipeline/vae_worker.py - Asynchronous VAE worker → VAE worker config: loads configuration
Matrix-Game-3/pipeline/vae_worker.py - VAE worker config → VAE implementation: configures VAE
Matrix-Game-3/pipeline/vae_config.py - Asynchronous VAE worker → VAE implementation: decodes chunks
Matrix-Game-3/pipeline/vae_worker.py - Asynchronous VAE worker → Generated video: saves video
Matrix-Game-3/pipeline/vae_worker.py - Player / operator → Scene evaluation CLI: requests evaluation
- Scene evaluation CLI → WorldScore evaluation: calls evaluator
Matrix-Game-1/GameWorldScore/evaluate_per_scene.py - WorldScore evaluation → World quality metrics: computes metrics
Matrix-Game-1/GameWorldScore/GameWorld/__init__.py