How wgcyeo/worldmm works
WorldMM turns long-video captions and frames into episodic, semantic, and visual memories, then evaluates question answering over those memories with retrieval and reasoning models. The graph follows the documented build-and-evaluate workflow and the sampled orchestration and wrapper code. Evidence for some memory and evaluation links is limited to README statements or source-index references; the separate bundled HippoRAG engine is shown only where its source index supports connections.
Main entry points: build_memory.py Memory Builder · eval.py QA Evaluation · extract_episodic_triples.py Episodic Memory · embedding_wrapper.py Embedding Router
Components 19
Video Preparation
- Video Captions
- Video Benchmarks
Memory Construction
- Memory Builder
preprocess/build_memory.py - Episodic Memory
preprocess/episodic_memory/extract_episodic_triples.py - Episodic OpenIE
src/worldmm/memory/episodic/openie.py - Semantic Extraction
preprocess/semantic_memory/extract_semantic_triples.py - Semantic Consolidation
preprocess/semantic_memory/consolidate_semantic_memory.py - Visual Memory
preprocess/visual_memory/extract_visual_features.py
Model Services
- Embedding Router
src/worldmm/embedding/embedding_wrapper.py - VLM2Vec Encoder
src/worldmm/embedding/vlm2vecv2.py - VLM Processor
src/worldmm/embedding/VLM2Vec/src/model/processor.py - LLM Wrapper
src/worldmm/llm/llm_wrapper.py - Prompt Manager
src/worldmm/llm/prompt_template_manager.py
Retrieval and Evaluation
- QA Evaluation
eval/eval.py - EgoLife Evaluation
eval/eval_egolife.py - HippoRAG Engine
src/HippoRAG/src/hipporag/HippoRAG.py - Embedding Store
src/HippoRAG/src/hipporag/embedding_store.py - Retrieval Metrics
src/HippoRAG/src/hipporag/evaluation/retrieval_eval.py
Other components
- Researcher
Connections 25
- Researcher → Memory Builder: builds memories
- Video Captions → Memory Builder: provides input
preprocess/build_memory.py - Memory Builder → Episodic Memory: runs extraction
preprocess/build_memory.py - Memory Builder → Semantic Extraction: runs extraction
preprocess/build_memory.py - Memory Builder → Semantic Consolidation: runs consolidation
preprocess/build_memory.py - Memory Builder → Visual Memory: runs extraction
preprocess/build_memory.py - Episodic Memory → Episodic OpenIE: uses OpenIE
preprocess/episodic_memory/extract_episodic_triples.py - Semantic Extraction → Episodic Memory: reads triples
preprocess/semantic_memory/extract_semantic_triples.py - Embedding Router → VLM2Vec Encoder: routes vision
src/worldmm/embedding/embedding_wrapper.py - VLM2Vec Encoder → VLM Processor: uses processor
- Visual Memory → Embedding Router: encodes video
preprocess/visual_memory/extract_visual_features.py - LLM Wrapper → Prompt Manager: uses templates
src/worldmm/llm/__init__.py - Researcher → QA Evaluation: evaluates questions
- Researcher → EgoLife Evaluation: evaluates EgoLife
- QA Evaluation → Episodic Memory: retrieves memory
- QA Evaluation → Semantic Extraction: retrieves memory
- QA Evaluation → Visual Memory: retrieves memory
- QA Evaluation → Embedding Router: uses retrieval model
eval/eval.py - QA Evaluation → LLM Wrapper: uses reasoning model
eval/eval.py - EgoLife Evaluation → Embedding Router: uses retrieval model
eval/eval_egolife.py - EgoLife Evaluation → LLM Wrapper: uses reasoning model
eval/eval_egolife.py - Video Benchmarks → Memory Builder: supports benchmark
- HippoRAG Engine → Episodic OpenIE: uses OpenIE
src/HippoRAG/src/hipporag/HippoRAG.py - HippoRAG Engine → Embedding Store: uses store
src/HippoRAG/src/hipporag/HippoRAG.py - HippoRAG Engine → Retrieval Metrics: uses metrics
src/HippoRAG/src/hipporag/HippoRAG.py