# wgcyeo/worldmm architecture

> Architecture diagram of the GitHub repository wgcyeo/worldmm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wgcyeo/worldmm
- Repository: https://github.com/wgcyeo/worldmm

## Overview

WorldMM turns long-video captions and frames into episodic, semantic, and visual memories, then evaluates question answering over those memories with retrieval and reasoning models. The graph follows the documented build-and-evaluate workflow and the sampled orchestration and wrapper code. Evidence for some memory and evaluation links is limited to README statements or source-index references; the separate bundled HippoRAG engine is shown only where its source index supports connections.

Main entry points: [`preprocess/build_memory.py`](https://github.com/wgcyeo/worldmm/blob/main/preprocess/build_memory.py) (Memory Builder), [`eval/eval.py`](https://github.com/wgcyeo/worldmm/blob/main/eval/eval.py) (QA Evaluation), [`preprocess/episodic_memory/extract_episodic_triples.py`](https://github.com/wgcyeo/worldmm/blob/main/preprocess/episodic_memory/extract_episodic_triples.py) (Episodic Memory), [`src/worldmm/embedding/embedding_wrapper.py`](https://github.com/wgcyeo/worldmm/blob/main/src/worldmm/embedding/embedding_wrapper.py) (Embedding Router)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_prep["Video Preparation"]
  node_captions["Video Captions"]
  node_video_bench["Video Benchmarks"]
end

subgraph group_memory["Memory Construction"]
  node_builder["Memory Builder<br/>[build_memory.py]"]
  node_episodic[("Episodic Memory")]
  node_episodic_core["Episodic OpenIE<br/>[openie.py]"]
  node_semantic_extract["Semantic Extraction"]
  node_semantic_consolidate["Semantic Consolidation"]
  node_visual[("Visual Memory")]
end

subgraph group_models["Model Services"]
  node_embeddings["Embedding Router"]
  node_vlm["VLM2Vec Encoder<br/>[vlm2vecv2.py]"]
  node_processor["VLM Processor<br/>[processor.py]"]
  node_llm["LLM Wrapper<br/>[llm_wrapper.py]"]
  node_prompts["Prompt Manager"]
end

subgraph group_reasoning["Retrieval and Evaluation"]
  node_qa["QA Evaluation<br/>[eval.py]"]
  node_egolife_qa["EgoLife Evaluation<br/>[eval_egolife.py]"]
  node_hipporag["HippoRAG Engine<br/>[HippoRAG.py]"]
  node_index[("Embedding Store<br/>[embedding_store.py]")]
  node_retrieval_eval["Retrieval Metrics<br/>[retrieval_eval.py]"]
end

node_user(("Researcher"))

node_user -->|"builds memories"| node_builder
node_captions -->|"provides input"| node_builder
node_builder -->|"runs extraction"| node_episodic
node_builder -->|"runs extraction"| node_semantic_extract
node_builder -->|"runs consolidation"| node_semantic_consolidate
node_builder -->|"runs extraction"| node_visual
node_episodic -->|"uses OpenIE"| node_episodic_core
node_semantic_extract -->|"reads triples"| node_episodic
node_embeddings -->|"routes vision"| node_vlm
node_vlm -->|"uses processor"| node_processor
node_visual -->|"encodes video"| node_embeddings
node_llm -->|"uses templates"| node_prompts
node_user -->|"evaluates questions"| node_qa
node_user -->|"evaluates EgoLife"| node_egolife_qa
node_qa -->|"retrieves memory"| node_episodic
node_qa -->|"retrieves memory"| node_semantic_extract
node_qa -->|"retrieves memory"| node_visual
node_qa -->|"uses retrieval model"| node_embeddings
node_qa -->|"uses reasoning model"| node_llm
node_egolife_qa -->|"uses retrieval model"| node_embeddings
node_egolife_qa -->|"uses reasoning model"| node_llm
node_video_bench -->|"supports benchmark"| node_builder
node_hipporag -->|"uses OpenIE"| node_episodic_core
node_hipporag -->|"uses store"| node_index
node_hipporag -->|"uses metrics"| node_retrieval_eval

click node_builder "https://github.com/wgcyeo/worldmm/blob/main/preprocess/build_memory.py"
click node_episodic "https://github.com/wgcyeo/worldmm/blob/main/preprocess/episodic_memory/extract_episodic_triples.py"
click node_episodic_core "https://github.com/wgcyeo/worldmm/blob/main/src/worldmm/memory/episodic/openie.py"
click node_semantic_extract "https://github.com/wgcyeo/worldmm/blob/main/preprocess/semantic_memory/extract_semantic_triples.py"
click node_semantic_consolidate "https://github.com/wgcyeo/worldmm/blob/main/preprocess/semantic_memory/consolidate_semantic_memory.py"
click node_visual "https://github.com/wgcyeo/worldmm/blob/main/preprocess/visual_memory/extract_visual_features.py"
click node_embeddings "https://github.com/wgcyeo/worldmm/blob/main/src/worldmm/embedding/embedding_wrapper.py"
click node_vlm "https://github.com/wgcyeo/worldmm/blob/main/src/worldmm/embedding/vlm2vecv2.py"
click node_processor "https://github.com/wgcyeo/worldmm/blob/main/src/worldmm/embedding/VLM2Vec/src/model/processor.py"
click node_llm "https://github.com/wgcyeo/worldmm/blob/main/src/worldmm/llm/llm_wrapper.py"
click node_prompts "https://github.com/wgcyeo/worldmm/blob/main/src/worldmm/llm/prompt_template_manager.py"
click node_qa "https://github.com/wgcyeo/worldmm/blob/main/eval/eval.py"
click node_egolife_qa "https://github.com/wgcyeo/worldmm/blob/main/eval/eval_egolife.py"
click node_hipporag "https://github.com/wgcyeo/worldmm/blob/main/src/HippoRAG/src/hipporag/HippoRAG.py"
click node_index "https://github.com/wgcyeo/worldmm/blob/main/src/HippoRAG/src/hipporag/embedding_store.py"
click node_retrieval_eval "https://github.com/wgcyeo/worldmm/blob/main/src/HippoRAG/src/hipporag/evaluation/retrieval_eval.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_captions,node_video_bench toneBlue
class node_builder,node_episodic,node_episodic_core,node_semantic_extract,node_semantic_consolidate,node_visual toneAmber
class node_embeddings,node_vlm,node_processor,node_llm,node_prompts toneMint
class node_qa,node_egolife_qa,node_hipporag,node_index,node_retrieval_eval toneRose
class node_user toneIndigo
```

## Components

### Video Preparation

- **Video Captions**
- **Video Benchmarks**

### Memory Construction

- [Memory Builder](https://github.com/wgcyeo/worldmm/blob/main/preprocess/build_memory.py): `preprocess/build_memory.py`
- [Episodic Memory](https://github.com/wgcyeo/worldmm/blob/main/preprocess/episodic_memory/extract_episodic_triples.py): `preprocess/episodic_memory/extract_episodic_triples.py`
- [Episodic OpenIE](https://github.com/wgcyeo/worldmm/blob/main/src/worldmm/memory/episodic/openie.py): `src/worldmm/memory/episodic/openie.py`
- [Semantic Extraction](https://github.com/wgcyeo/worldmm/blob/main/preprocess/semantic_memory/extract_semantic_triples.py): `preprocess/semantic_memory/extract_semantic_triples.py`
- [Semantic Consolidation](https://github.com/wgcyeo/worldmm/blob/main/preprocess/semantic_memory/consolidate_semantic_memory.py): `preprocess/semantic_memory/consolidate_semantic_memory.py`
- [Visual Memory](https://github.com/wgcyeo/worldmm/blob/main/preprocess/visual_memory/extract_visual_features.py): `preprocess/visual_memory/extract_visual_features.py`

### Model Services

- [Embedding Router](https://github.com/wgcyeo/worldmm/blob/main/src/worldmm/embedding/embedding_wrapper.py): `src/worldmm/embedding/embedding_wrapper.py`
- [VLM2Vec Encoder](https://github.com/wgcyeo/worldmm/blob/main/src/worldmm/embedding/vlm2vecv2.py): `src/worldmm/embedding/vlm2vecv2.py`
- [VLM Processor](https://github.com/wgcyeo/worldmm/blob/main/src/worldmm/embedding/VLM2Vec/src/model/processor.py): `src/worldmm/embedding/VLM2Vec/src/model/processor.py`
- [LLM Wrapper](https://github.com/wgcyeo/worldmm/blob/main/src/worldmm/llm/llm_wrapper.py): `src/worldmm/llm/llm_wrapper.py`
- [Prompt Manager](https://github.com/wgcyeo/worldmm/blob/main/src/worldmm/llm/prompt_template_manager.py): `src/worldmm/llm/prompt_template_manager.py`

### Retrieval and Evaluation

- [QA Evaluation](https://github.com/wgcyeo/worldmm/blob/main/eval/eval.py): `eval/eval.py`
- [EgoLife Evaluation](https://github.com/wgcyeo/worldmm/blob/main/eval/eval_egolife.py): `eval/eval_egolife.py`
- [HippoRAG Engine](https://github.com/wgcyeo/worldmm/blob/main/src/HippoRAG/src/hipporag/HippoRAG.py): `src/HippoRAG/src/hipporag/HippoRAG.py`
- [Embedding Store](https://github.com/wgcyeo/worldmm/blob/main/src/HippoRAG/src/hipporag/embedding_store.py): `src/HippoRAG/src/hipporag/embedding_store.py`
- [Retrieval Metrics](https://github.com/wgcyeo/worldmm/blob/main/src/HippoRAG/src/hipporag/evaluation/retrieval_eval.py): `src/HippoRAG/src/hipporag/evaluation/retrieval_eval.py`

### Other components

- **Researcher**

## Connections

- Researcher → Memory Builder: builds memories
- Video Captions → Memory Builder: provides input (evidence: `preprocess/build_memory.py`)
- Memory Builder → Episodic Memory: runs extraction (evidence: `preprocess/build_memory.py`)
- Memory Builder → Semantic Extraction: runs extraction (evidence: `preprocess/build_memory.py`)
- Memory Builder → Semantic Consolidation: runs consolidation (evidence: `preprocess/build_memory.py`)
- Memory Builder → Visual Memory: runs extraction (evidence: `preprocess/build_memory.py`)
- Episodic Memory → Episodic OpenIE: uses OpenIE (evidence: `preprocess/episodic_memory/extract_episodic_triples.py`)
- Semantic Extraction → Episodic Memory: reads triples (evidence: `preprocess/semantic_memory/extract_semantic_triples.py`)
- Embedding Router → VLM2Vec Encoder: routes vision (evidence: `src/worldmm/embedding/embedding_wrapper.py`)
- VLM2Vec Encoder → VLM Processor: uses processor
- Visual Memory → Embedding Router: encodes video (evidence: `preprocess/visual_memory/extract_visual_features.py`)
- LLM Wrapper → Prompt Manager: uses templates (evidence: `src/worldmm/llm/__init__.py`)
- Researcher → QA Evaluation: evaluates questions
- Researcher → EgoLife Evaluation: evaluates EgoLife
- QA Evaluation → Episodic Memory: retrieves memory
- QA Evaluation → Semantic Extraction: retrieves memory
- QA Evaluation → Visual Memory: retrieves memory
- QA Evaluation → Embedding Router: uses retrieval model (evidence: `eval/eval.py`)
- QA Evaluation → LLM Wrapper: uses reasoning model (evidence: `eval/eval.py`)
- EgoLife Evaluation → Embedding Router: uses retrieval model (evidence: `eval/eval_egolife.py`)
- EgoLife Evaluation → LLM Wrapper: uses reasoning model (evidence: `eval/eval_egolife.py`)
- Video Benchmarks → Memory Builder: supports benchmark
- HippoRAG Engine → Episodic OpenIE: uses OpenIE (evidence: `src/HippoRAG/src/hipporag/HippoRAG.py`)
- HippoRAG Engine → Embedding Store: uses store (evidence: `src/HippoRAG/src/hipporag/HippoRAG.py`)
- HippoRAG Engine → Retrieval Metrics: uses metrics (evidence: `src/HippoRAG/src/hipporag/HippoRAG.py`)
