How tomljm/mugen-vr works
MUGen-VR combines multimodal feature encoding, condition fusion, reference retrieval, and AnyFlow video generation. The principal inference flow takes text and media conditions, retrieves scored video references, builds condition tokens, and injects them into AnyFlow prompt embeddings to produce video frames. The README also documents feature-store preparation, training, and evaluation as important supporting capabilities. Some integration modules and scripts are represented from documented responsibilities; their wiring is omitted where sampled source does not establish it.
Main entry points: retrieve_then_generate.py Retrieve Then Generate · feature_store.py Feature Store · conditioner.py Multimodal Conditioner · imagebind_encoder.py ImageBind Encoders
Components 17
Conditioned Generation
- Audio Prompt Parser
src/mugen/conditioning/audio_prompt_parser.py - Retrieve Then Generate
src/mugen/generation/retrieve_then_generate.py - Cross-Modal Retriever
src/mugen/retrieval/retriever.py - Feature Index
src/mugen/retrieval/feature_index.py - Multimodal Conditioner
src/mugen/generation/conditioner.py - Condition Bundle
src/mugen/common/interfaces.py - AnyFlow Generator
src/mugen/generation/generators/anyflow_generator.py
Multimodal Representations
- ImageBind Encoders
src/mugen/encoders/imagebind_encoder.py - InternVideo Encoder
src/mugen/encoders/internvideo_encoder.py - Hierarchical Condition Fusion
src/mugen/fusion/fusion_module.py
Features and Training
- Dataset and Manifest
src/mugen/data/dataset.py - Feature Store
src/mugen/data/feature_store.py - Training Objectives
src/mugen/training/losses.py
Evaluation
- Evaluation Runner
src/mugen/evaluation/runner.py - VBench Adapter
src/mugen/evaluation/vbench_adapter.py - Generated Video
Other components
- User
Connections 17
- User → Retrieve Then Generate: provides conditions
- User → Audio Prompt Parser: provides prompt
- Audio Prompt Parser → Retrieve Then Generate: parses audio style
- Retrieve Then Generate → Cross-Modal Retriever: retrieves references
- Cross-Modal Retriever → Feature Index: searches
- Retrieve Then Generate → Multimodal Conditioner: builds condition tokens
- ImageBind Encoders → Hierarchical Condition Fusion: encodes text image audio
- InternVideo Encoder → Feature Store: writes video features
- ImageBind Encoders → Feature Store: writes modality features
- Hierarchical Condition Fusion → Multimodal Conditioner: fuses modalities
- Multimodal Conditioner → Condition Bundle: projects tokens
- Condition Bundle → AnyFlow Generator: supplies prompt embeddings
- Retrieve Then Generate → AnyFlow Generator: generates video
- Dataset and Manifest → Feature Store: loads and stores features
- Feature Store → Training Objectives: provides features
- Generated Video → Evaluation Runner: evaluated by
- Evaluation Runner → VBench Adapter: requests metrics