# skyworkai/matrix-game architecture

> Architecture diagram of the GitHub repository skyworkai/matrix-game, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/skyworkai/matrix-game
- Repository: https://github.com/skyworkai/matrix-game

## Overview

Matrix-Game is a three-generation family of interactive world models that generates video conditioned on images/prompts and player controls. The graph follows the sampled Matrix-Game-2 image-conditioned and streaming paths, then the Matrix-Game-3 generation and asynchronous VAE decode path; Matrix-Game-1 inference and its scoring capabilities remain represented as a distinct branch. Several relationships and capabilities come from README or SOURCE INDEX references rather than complete excerpts, so uncertain links are omitted.

Main entry points: [`Matrix-Game-2/pipeline/causal_inference.py`](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-2/pipeline/causal_inference.py) (Causal generation pipeline), [`Matrix-Game-1/inference_bench.py`](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-1/inference_bench.py) (Image-to-video inference), [`Matrix-Game-1/parallel_infer.py`](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-1/parallel_infer.py) (Parallel inference), [`Matrix-Game-3/pipeline/inference_interactive_pipeline.py`](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-3/pipeline/inference_interactive_pipeline.py) (Interactive streaming pipeline)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g_entry["User workflows"]
end

subgraph group_g_mg1["Matrix-Game 1"]
  node_mg1_infer["Image-to-video inference<br/>[inference_bench.py]"]
  node_mg1_enc["Text and image encoders"]
  node_mg1_dit["Diffusion transformer<br/>[models_i2v.py]"]
  node_mg1_vae["Video VAE<br/>[matrixgame_vae.py]"]
  node_mg1_dist["Parallel inference<br/>[parallel_infer.py]"]
  node_mg1_vis["Video visualization<br/>[visualize.py]"]
end

subgraph group_g_mg2["Matrix-Game 2"]
  node_mg2_infer["Interactive inference<br/>[inference.py]"]
  node_mg2_stream["Streaming inference"]
  node_mg2_pipe["Causal generation pipeline"]
  node_mg2_wrap["Model and scheduler wrapper<br/>[wan_wrapper.py]"]
  node_mg2_model["Video diffusion model<br/>[model.py]"]
  node_mg2_causal["Causal video model<br/>[causal_model.py]"]
  node_mg2_action["Action conditioning<br/>[action_module.py]"]
  node_mg2_vae["Wan video VAE<br/>[vae.py]"]
  node_mg2_decode["Streaming VAE decode<br/>[vae_block3.py]"]
  node_mg2_overlay["Control overlays<br/>[visualize.py]"]
  node_mg2_scheduler["Flow-match solver"]
end

subgraph group_g_mg3["Matrix-Game 3"]
  node_mg3_generate["Generation entrypoint<br/>[generate.py]"]
  node_mg3_pipe["Interactive streaming pipeline"]
  node_mg3_batch["Standard generation pipeline"]
  node_mg3_model["Video diffusion model<br/>[model.py]"]
  node_mg3_vae["VAE implementation<br/>[vae2_2.py]"]
  node_mg3_config["VAE worker config<br/>[vae_config.py]"]
  node_mg3_worker["Asynchronous VAE worker<br/>[vae_worker.py]"]
  node_mg3_parallel["Sequence parallelism"]
end

subgraph group_g_score["Evaluation"]
  node_mg1_score["WorldScore evaluation<br/>[__init__.py]"]
  node_mg1_score_cli["Scene evaluation CLI"]
  node_mg1_metrics["World quality metrics"]
end

node_user(("Player / operator"))
node_video["Generated video"]
node_latent_queue["Latent queue"]

node_user -.->|"provides inputs"| node_mg1_infer
node_mg1_infer -->|"loads encoders"| node_mg1_enc
node_mg1_infer -->|"loads transformer"| node_mg1_dit
node_mg1_infer -->|"loads VAE"| node_mg1_vae
node_mg1_infer -->|"renders video"| node_mg1_vis
node_mg1_infer -.->|"generates"| node_video
node_mg1_dist -->|"loads encoders"| node_mg1_enc
node_mg1_dist -->|"loads VAE"| node_mg1_vae
node_mg1_dist -->|"loads transformer"| node_mg1_dit
node_mg1_dist -->|"renders video"| node_mg1_vis
node_user -.->|"sets conditions"| node_mg2_infer
node_user -.->|"sets conditions"| node_mg2_stream
node_mg2_infer -->|"runs pipeline"| node_mg2_pipe
node_mg2_stream -->|"runs pipeline"| node_mg2_pipe
node_mg2_pipe -->|"constructs generator"| node_mg2_wrap
node_mg2_wrap -->|"uses model"| node_mg2_model
node_mg2_wrap -->|"uses causal model"| node_mg2_causal
node_mg2_model -->|"applies actions"| node_mg2_action
node_mg2_pipe -.->|"denoises with"| node_mg2_scheduler
node_mg2_pipe -.->|"decodes frames"| node_mg2_decode
node_mg2_pipe -->|"renders controls"| node_mg2_overlay
node_mg2_pipe -.->|"produces video"| node_video
node_mg3_generate -->|"selects pipeline"| node_mg3_pipe
node_mg3_generate -->|"selects pipeline"| node_mg3_batch
node_user -.->|"provides prompts / controls"| node_mg3_generate
node_mg3_pipe -->|"runs denoising"| node_mg3_model
node_mg3_batch -->|"runs denoising"| node_mg3_model
node_mg3_pipe -->|"uses parallelism"| node_mg3_parallel
node_mg3_batch -->|"uses parallelism"| node_mg3_parallel
node_mg3_pipe -->|"enqueues latents"| node_latent_queue
node_mg3_batch -->|"enqueues latents"| node_latent_queue
node_latent_queue -->|"delivers latents"| node_mg3_worker
node_mg3_worker -->|"loads configuration"| node_mg3_config
node_mg3_config -->|"configures VAE"| node_mg3_vae
node_mg3_worker -->|"decodes chunks"| node_mg3_vae
node_mg3_worker -->|"saves video"| node_video
node_user -.->|"requests evaluation"| node_mg1_score_cli
node_mg1_score_cli -->|"calls evaluator"| node_mg1_score
node_mg1_score -->|"computes metrics"| node_mg1_metrics

click node_mg1_infer "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-1/inference_bench.py"
click node_mg1_enc "https://github.com/skyworkai/matrix-game/tree/main/Matrix-Game-1/matrixgame/encoder_variants"
click node_mg1_dit "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-1/matrixgame/model_variants/matrixgame_dit_src/models_i2v.py"
click node_mg1_vae "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-1/matrixgame/vae_variants/matrixgame_vae.py"
click node_mg1_dist "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-1/parallel_infer.py"
click node_mg1_vis "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-1/tools/visualize.py"
click node_mg1_score "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-1/GameWorldScore/GameWorld/__init__.py"
click node_mg1_score_cli "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-1/GameWorldScore/evaluate_per_scene.py"
click node_mg1_metrics "https://github.com/skyworkai/matrix-game/tree/main/Matrix-Game-1/GameWorldScore/GameWorld"
click node_mg2_infer "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-2/inference.py"
click node_mg2_stream "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-2/inference_streaming.py"
click node_mg2_pipe "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-2/pipeline/causal_inference.py"
click node_mg2_wrap "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-2/utils/wan_wrapper.py"
click node_mg2_model "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-2/wan/modules/model.py"
click node_mg2_causal "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-2/wan/modules/causal_model.py"
click node_mg2_action "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-2/wan/modules/action_module.py"
click node_mg2_vae "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-2/wan/modules/vae.py"
click node_mg2_decode "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-2/demo_utils/vae_block3.py"
click node_mg2_overlay "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-2/utils/visualize.py"
click node_mg2_scheduler "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-2/wan/utils/fm_solvers_unipc.py"
click node_mg3_generate "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-3/generate.py"
click node_mg3_pipe "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-3/pipeline/inference_interactive_pipeline.py"
click node_mg3_batch "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-3/pipeline/inference_pipeline.py"
click node_mg3_model "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-3/wan/modules/model.py"
click node_mg3_vae "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-3/wan/modules/vae2_2.py"
click node_mg3_config "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-3/pipeline/vae_config.py"
click node_mg3_worker "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-3/pipeline/vae_worker.py"
click node_mg3_parallel "https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-3/wan/distributed/sequence_parallel.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_mg1_infer,node_mg1_enc,node_mg1_dit,node_mg1_vae,node_mg1_dist,node_mg1_vis toneAmber
class node_mg2_infer,node_mg2_stream,node_mg2_pipe,node_mg2_wrap,node_mg2_model,node_mg2_causal,node_mg2_action,node_mg2_vae,node_mg2_decode,node_mg2_overlay,node_mg2_scheduler toneMint
class node_mg3_generate,node_mg3_pipe,node_mg3_batch,node_mg3_model,node_mg3_vae,node_mg3_config,node_mg3_worker,node_mg3_parallel,node_latent_queue toneRose
class node_mg1_score,node_mg1_score_cli,node_mg1_metrics,node_user,node_video toneIndigo
```

## Components

### Matrix-Game 1

- [Image-to-video inference](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-1/inference_bench.py): `Matrix-Game-1/inference_bench.py`
- [Text and image encoders](https://github.com/skyworkai/matrix-game/tree/main/Matrix-Game-1/matrixgame/encoder_variants): `Matrix-Game-1/matrixgame/encoder_variants`
- [Diffusion transformer](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-1/matrixgame/model_variants/matrixgame_dit_src/models_i2v.py): `Matrix-Game-1/matrixgame/model_variants/matrixgame_dit_src/models_i2v.py`
- [Video VAE](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-1/matrixgame/vae_variants/matrixgame_vae.py): `Matrix-Game-1/matrixgame/vae_variants/matrixgame_vae.py`
- [Parallel inference](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-1/parallel_infer.py): `Matrix-Game-1/parallel_infer.py`
- [Video visualization](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-1/tools/visualize.py): `Matrix-Game-1/tools/visualize.py`

### Matrix-Game 2

- [Interactive inference](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-2/inference.py): `Matrix-Game-2/inference.py`
- [Streaming inference](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-2/inference_streaming.py): `Matrix-Game-2/inference_streaming.py`
- [Causal generation pipeline](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-2/pipeline/causal_inference.py): `Matrix-Game-2/pipeline/causal_inference.py`
- [Model and scheduler wrapper](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-2/utils/wan_wrapper.py): `Matrix-Game-2/utils/wan_wrapper.py`
- [Video diffusion model](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-2/wan/modules/model.py): `Matrix-Game-2/wan/modules/model.py`
- [Causal video model](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-2/wan/modules/causal_model.py): `Matrix-Game-2/wan/modules/causal_model.py`
- [Action conditioning](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-2/wan/modules/action_module.py): `Matrix-Game-2/wan/modules/action_module.py`
- [Wan video VAE](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-2/wan/modules/vae.py): `Matrix-Game-2/wan/modules/vae.py`
- [Streaming VAE decode](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-2/demo_utils/vae_block3.py): `Matrix-Game-2/demo_utils/vae_block3.py`
- [Control overlays](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-2/utils/visualize.py): `Matrix-Game-2/utils/visualize.py`
- [Flow-match solver](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-2/wan/utils/fm_solvers_unipc.py): `Matrix-Game-2/wan/utils/fm_solvers_unipc.py`

### Matrix-Game 3

- [Generation entrypoint](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-3/generate.py): `Matrix-Game-3/generate.py`
- [Interactive streaming pipeline](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-3/pipeline/inference_interactive_pipeline.py): `Matrix-Game-3/pipeline/inference_interactive_pipeline.py`
- [Standard generation pipeline](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-3/pipeline/inference_pipeline.py): `Matrix-Game-3/pipeline/inference_pipeline.py`
- [Video diffusion model](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-3/wan/modules/model.py): `Matrix-Game-3/wan/modules/model.py`
- [VAE implementation](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-3/wan/modules/vae2_2.py): `Matrix-Game-3/wan/modules/vae2_2.py`
- [VAE worker config](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-3/pipeline/vae_config.py): `Matrix-Game-3/pipeline/vae_config.py`
- [Asynchronous VAE worker](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-3/pipeline/vae_worker.py): `Matrix-Game-3/pipeline/vae_worker.py`
- [Sequence parallelism](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-3/wan/distributed/sequence_parallel.py): `Matrix-Game-3/wan/distributed/sequence_parallel.py`

### Evaluation

- [WorldScore evaluation](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-1/GameWorldScore/GameWorld/__init__.py): `Matrix-Game-1/GameWorldScore/GameWorld/__init__.py`
- [Scene evaluation CLI](https://github.com/skyworkai/matrix-game/blob/main/Matrix-Game-1/GameWorldScore/evaluate_per_scene.py): `Matrix-Game-1/GameWorldScore/evaluate_per_scene.py`
- [World quality metrics](https://github.com/skyworkai/matrix-game/tree/main/Matrix-Game-1/GameWorldScore/GameWorld): `Matrix-Game-1/GameWorldScore/GameWorld`

### Other components

- **Player / operator**
- **Generated video**
- **Latent queue**

## Connections

- Player / operator → Image-to-video inference: provides inputs
- Image-to-video inference → Text and image encoders: loads encoders (evidence: `Matrix-Game-1/inference_bench.py`)
- Image-to-video inference → Diffusion transformer: loads transformer (evidence: `Matrix-Game-1/inference_bench.py`)
- Image-to-video inference → Video VAE: loads VAE (evidence: `Matrix-Game-1/inference_bench.py`)
- Image-to-video inference → Video visualization: renders video (evidence: `Matrix-Game-1/inference_bench.py`)
- Image-to-video inference → Generated video: generates
- Parallel inference → Text and image encoders: loads encoders (evidence: `Matrix-Game-1/parallel_infer.py`)
- Parallel inference → Video VAE: loads VAE (evidence: `Matrix-Game-1/parallel_infer.py`)
- Parallel inference → Diffusion transformer: loads transformer (evidence: `Matrix-Game-1/parallel_infer.py`)
- Parallel inference → Video visualization: renders video (evidence: `Matrix-Game-1/parallel_infer.py`)
- Player / operator → Interactive inference: sets conditions
- Player / operator → Streaming inference: sets conditions
- Interactive inference → Causal generation pipeline: runs pipeline (evidence: `Matrix-Game-2/inference.py`)
- Streaming inference → Causal generation pipeline: runs pipeline (evidence: `Matrix-Game-2/inference_streaming.py`)
- Causal generation pipeline → Model and scheduler wrapper: constructs generator (evidence: `Matrix-Game-2/pipeline/causal_inference.py`)
- Model and scheduler wrapper → Video diffusion model: uses model (evidence: `Matrix-Game-2/utils/wan_wrapper.py`)
- Model and scheduler wrapper → Causal video model: uses causal model (evidence: `Matrix-Game-2/utils/wan_wrapper.py`)
- Video diffusion model → Action conditioning: applies actions (evidence: `Matrix-Game-2/wan/modules/model.py`)
- Causal generation pipeline → Flow-match solver: denoises with
- Causal generation pipeline → Streaming VAE decode: decodes frames
- Causal generation pipeline → Control overlays: renders controls (evidence: `Matrix-Game-2/pipeline/causal_inference.py`)
- Causal generation pipeline → Generated video: produces video
- Generation entrypoint → Interactive streaming pipeline: selects pipeline (evidence: `Matrix-Game-3/generate.py`)
- Generation entrypoint → Standard generation pipeline: selects pipeline (evidence: `Matrix-Game-3/generate.py`)
- Player / operator → Generation entrypoint: provides prompts / controls
- Interactive streaming pipeline → Video diffusion model: runs denoising (evidence: `Matrix-Game-3/pipeline/inference_interactive_pipeline.py`)
- Standard generation pipeline → Video diffusion model: runs denoising (evidence: `Matrix-Game-3/pipeline/inference_pipeline.py`)
- Interactive streaming pipeline → Sequence parallelism: uses parallelism (evidence: `Matrix-Game-3/pipeline/inference_interactive_pipeline.py`)
- Standard generation pipeline → Sequence parallelism: uses parallelism (evidence: `Matrix-Game-3/pipeline/inference_pipeline.py`)
- Interactive streaming pipeline → Latent queue: enqueues latents (evidence: `Matrix-Game-3/pipeline/inference_interactive_pipeline.py`)
- Standard generation pipeline → Latent queue: enqueues latents (evidence: `Matrix-Game-3/pipeline/inference_pipeline.py`)
- Latent queue → Asynchronous VAE worker: delivers latents (evidence: `Matrix-Game-3/pipeline/vae_worker.py`)
- Asynchronous VAE worker → VAE worker config: loads configuration (evidence: `Matrix-Game-3/pipeline/vae_worker.py`)
- VAE worker config → VAE implementation: configures VAE (evidence: `Matrix-Game-3/pipeline/vae_config.py`)
- Asynchronous VAE worker → VAE implementation: decodes chunks (evidence: `Matrix-Game-3/pipeline/vae_worker.py`)
- Asynchronous VAE worker → Generated video: saves video (evidence: `Matrix-Game-3/pipeline/vae_worker.py`)
- Player / operator → Scene evaluation CLI: requests evaluation
- Scene evaluation CLI → WorldScore evaluation: calls evaluator (evidence: `Matrix-Game-1/GameWorldScore/evaluate_per_scene.py`)
- WorldScore evaluation → World quality metrics: computes metrics (evidence: `Matrix-Game-1/GameWorldScore/GameWorld/__init__.py`)
