# tomljm/mugen-vr architecture

> Architecture diagram of the GitHub repository tomljm/mugen-vr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tomljm/mugen-vr
- Repository: https://github.com/tomljm/mugen-vr

## Overview

MUGen-VR combines multimodal feature encoding, condition fusion, reference retrieval, and AnyFlow video generation. The principal inference flow takes text and media conditions, retrieves scored video references, builds condition tokens, and injects them into AnyFlow prompt embeddings to produce video frames. The README also documents feature-store preparation, training, and evaluation as important supporting capabilities. Some integration modules and scripts are represented from documented responsibilities; their wiring is omitted where sampled source does not establish it.

Main entry points: [`src/mugen/generation/retrieve_then_generate.py`](https://github.com/tomljm/mugen-vr/blob/main/src/mugen/generation/retrieve_then_generate.py) (Retrieve Then Generate), [`src/mugen/data/feature_store.py`](https://github.com/tomljm/mugen-vr/blob/main/src/mugen/data/feature_store.py) (Feature Store), [`src/mugen/generation/conditioner.py`](https://github.com/tomljm/mugen-vr/blob/main/src/mugen/generation/conditioner.py) (Multimodal Conditioner), [`src/mugen/encoders/imagebind_encoder.py`](https://github.com/tomljm/mugen-vr/blob/main/src/mugen/encoders/imagebind_encoder.py) (ImageBind Encoders)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inference["Conditioned Generation"]
  node_prompt_parser["Audio Prompt Parser"]
  node_orchestrator["Retrieve Then Generate"]
  node_retriever["Cross-Modal Retriever<br/>[retriever.py]"]
  node_feature_index[("Feature Index<br/>[feature_index.py]")]
  node_conditioner["Multimodal Conditioner<br/>[conditioner.py]"]
  node_condition_bundle["Condition Bundle<br/>[interfaces.py]"]
  node_anyflow["AnyFlow Generator"]
end

subgraph group_representations["Multimodal Representations"]
  node_imagebind["ImageBind Encoders"]
  node_internvideo["InternVideo Encoder"]
  node_fusion["Hierarchical Condition Fusion<br/>[fusion_module.py]"]
end

subgraph group_data["Features and Training"]
  node_dataset["Dataset and Manifest<br/>[dataset.py]"]
  node_feature_store[("Feature Store<br/>[feature_store.py]")]
  node_training["Training Objectives<br/>[losses.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_eval_runner["Evaluation Runner<br/>[runner.py]"]
  node_vbench["VBench Adapter<br/>[vbench_adapter.py]"]
  node_generated_video["Generated Video"]
end

node_user(("User"))

node_user -->|"provides conditions"| node_orchestrator
node_user -.->|"provides prompt"| node_prompt_parser
node_prompt_parser -.->|"parses audio style"| node_orchestrator
node_orchestrator -->|"retrieves references"| node_retriever
node_retriever -->|"searches"| node_feature_index
node_orchestrator -->|"builds condition tokens"| node_conditioner
node_imagebind -->|"encodes text image audio"| node_fusion
node_internvideo -.->|"writes video features"| node_feature_store
node_imagebind -.->|"writes modality features"| node_feature_store
node_fusion -->|"fuses modalities"| node_conditioner
node_conditioner -->|"projects tokens"| node_condition_bundle
node_condition_bundle -->|"supplies prompt embeddings"| node_anyflow
node_orchestrator -->|"generates video"| node_anyflow
node_dataset -.->|"loads and stores features"| node_feature_store
node_feature_store -.->|"provides features"| node_training
node_generated_video -->|"evaluated by"| node_eval_runner
node_eval_runner -->|"requests metrics"| node_vbench

click node_prompt_parser "https://github.com/tomljm/mugen-vr/blob/main/src/mugen/conditioning/audio_prompt_parser.py"
click node_orchestrator "https://github.com/tomljm/mugen-vr/blob/main/src/mugen/generation/retrieve_then_generate.py"
click node_retriever "https://github.com/tomljm/mugen-vr/blob/main/src/mugen/retrieval/retriever.py"
click node_feature_index "https://github.com/tomljm/mugen-vr/blob/main/src/mugen/retrieval/feature_index.py"
click node_conditioner "https://github.com/tomljm/mugen-vr/blob/main/src/mugen/generation/conditioner.py"
click node_condition_bundle "https://github.com/tomljm/mugen-vr/blob/main/src/mugen/common/interfaces.py"
click node_anyflow "https://github.com/tomljm/mugen-vr/blob/main/src/mugen/generation/generators/anyflow_generator.py"
click node_imagebind "https://github.com/tomljm/mugen-vr/blob/main/src/mugen/encoders/imagebind_encoder.py"
click node_internvideo "https://github.com/tomljm/mugen-vr/blob/main/src/mugen/encoders/internvideo_encoder.py"
click node_fusion "https://github.com/tomljm/mugen-vr/blob/main/src/mugen/fusion/fusion_module.py"
click node_dataset "https://github.com/tomljm/mugen-vr/blob/main/src/mugen/data/dataset.py"
click node_feature_store "https://github.com/tomljm/mugen-vr/blob/main/src/mugen/data/feature_store.py"
click node_training "https://github.com/tomljm/mugen-vr/blob/main/src/mugen/training/losses.py"
click node_eval_runner "https://github.com/tomljm/mugen-vr/blob/main/src/mugen/evaluation/runner.py"
click node_vbench "https://github.com/tomljm/mugen-vr/blob/main/src/mugen/evaluation/vbench_adapter.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_prompt_parser,node_orchestrator,node_retriever,node_feature_index,node_conditioner,node_condition_bundle,node_anyflow,node_user toneBlue
class node_imagebind,node_internvideo,node_fusion toneAmber
class node_dataset,node_feature_store,node_training toneMint
class node_eval_runner,node_vbench,node_generated_video toneRose
```

## Components

### Conditioned Generation

- [Audio Prompt Parser](https://github.com/tomljm/mugen-vr/blob/main/src/mugen/conditioning/audio_prompt_parser.py): `src/mugen/conditioning/audio_prompt_parser.py`
- [Retrieve Then Generate](https://github.com/tomljm/mugen-vr/blob/main/src/mugen/generation/retrieve_then_generate.py): `src/mugen/generation/retrieve_then_generate.py`
- [Cross-Modal Retriever](https://github.com/tomljm/mugen-vr/blob/main/src/mugen/retrieval/retriever.py): `src/mugen/retrieval/retriever.py`
- [Feature Index](https://github.com/tomljm/mugen-vr/blob/main/src/mugen/retrieval/feature_index.py): `src/mugen/retrieval/feature_index.py`
- [Multimodal Conditioner](https://github.com/tomljm/mugen-vr/blob/main/src/mugen/generation/conditioner.py): `src/mugen/generation/conditioner.py`
- [Condition Bundle](https://github.com/tomljm/mugen-vr/blob/main/src/mugen/common/interfaces.py): `src/mugen/common/interfaces.py`
- [AnyFlow Generator](https://github.com/tomljm/mugen-vr/blob/main/src/mugen/generation/generators/anyflow_generator.py): `src/mugen/generation/generators/anyflow_generator.py`

### Multimodal Representations

- [ImageBind Encoders](https://github.com/tomljm/mugen-vr/blob/main/src/mugen/encoders/imagebind_encoder.py): `src/mugen/encoders/imagebind_encoder.py`
- [InternVideo Encoder](https://github.com/tomljm/mugen-vr/blob/main/src/mugen/encoders/internvideo_encoder.py): `src/mugen/encoders/internvideo_encoder.py`
- [Hierarchical Condition Fusion](https://github.com/tomljm/mugen-vr/blob/main/src/mugen/fusion/fusion_module.py): `src/mugen/fusion/fusion_module.py`

### Features and Training

- [Dataset and Manifest](https://github.com/tomljm/mugen-vr/blob/main/src/mugen/data/dataset.py): `src/mugen/data/dataset.py`
- [Feature Store](https://github.com/tomljm/mugen-vr/blob/main/src/mugen/data/feature_store.py): `src/mugen/data/feature_store.py`
- [Training Objectives](https://github.com/tomljm/mugen-vr/blob/main/src/mugen/training/losses.py): `src/mugen/training/losses.py`

### Evaluation

- [Evaluation Runner](https://github.com/tomljm/mugen-vr/blob/main/src/mugen/evaluation/runner.py): `src/mugen/evaluation/runner.py`
- [VBench Adapter](https://github.com/tomljm/mugen-vr/blob/main/src/mugen/evaluation/vbench_adapter.py): `src/mugen/evaluation/vbench_adapter.py`
- **Generated Video**

### Other components

- **User**

## Connections

- User → Retrieve Then Generate: provides conditions
- User → Audio Prompt Parser: provides prompt
- Audio Prompt Parser → Retrieve Then Generate: parses audio style
- Retrieve Then Generate → Cross-Modal Retriever: retrieves references
- Cross-Modal Retriever → Feature Index: searches
- Retrieve Then Generate → Multimodal Conditioner: builds condition tokens
- ImageBind Encoders → Hierarchical Condition Fusion: encodes text image audio
- InternVideo Encoder → Feature Store: writes video features
- ImageBind Encoders → Feature Store: writes modality features
- Hierarchical Condition Fusion → Multimodal Conditioner: fuses modalities
- Multimodal Conditioner → Condition Bundle: projects tokens
- Condition Bundle → AnyFlow Generator: supplies prompt embeddings
- Retrieve Then Generate → AnyFlow Generator: generates video
- Dataset and Manifest → Feature Store: loads and stores features
- Feature Store → Training Objectives: provides features
- Generated Video → Evaluation Runner: evaluated by
- Evaluation Runner → VBench Adapter: requests metrics