# sitonggong/grove architecture

> Architecture diagram of the GitHub repository sitonggong/grove, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sitonggong/grove
- Repository: https://github.com/sitonggong/grove

## Overview

GROVE is a training-free streaming-video memory for both reactive history QA and proactive assistance. Its documented construction flow captions incoming video, segments activity into episodes, extracts facts and aggregates recurring topics, then keeps the resulting memory and indexes queryable. Reactive agents choose retrieval skills over multiple rounds; proactive agents use the current situation to assemble evidence. The tree also contains benchmark-specific processing and inference entry points. Relationships among those unsampled entry points are not expanded; edges below rely on README statements or shown source excerpts.

Main entry points: [`hypervideo/structure.py`](https://github.com/sitonggong/grove/blob/main/hypervideo/structure.py) (Stratified memory), [`hypervideo/retrieval/hierarchical_retriever.py`](https://github.com/sitonggong/grove/blob/main/hypervideo/retrieval/hierarchical_retriever.py) (Hierarchical retrieval), [`hypervideo/extractors/hypergraph_builder.py`](https://github.com/sitonggong/grove/blob/main/hypervideo/extractors/hypergraph_builder.py) (Hypergraph builder), [`hypervideo/sp_agentic_qa_v2.py`](https://github.com/sitonggong/grove/blob/main/hypervideo/sp_agentic_qa_v2.py) (Reactive QA agent)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_construction["Memory Construction"]
  node_caption["Perceptual captioning"]
  node_segment["Episode segmentation"]
  node_builder["Hypergraph builder"]
  node_topic["Topic aggregation"]
end

subgraph group_memory["Memory and Retrieval"]
  node_memorygraph[("Stratified memory<br/>[structure.py]")]
  node_dualindex[("Dual index<br/>[dual_index.py]")]
  node_retriever["Hierarchical retrieval"]
  node_skills["Retrieval skills<br/>[__init__.py]"]
  node_visual["Visual indexing<br/>[visual_indexing.py]"]
end

subgraph group_assist["Assistant Workflows"]
  node_qa["Reactive QA agent"]
  node_hierqa["Hierarchical QA<br/>[sp_hierarchical.py]"]
  node_services["Proactive service"]
  node_estp["Proactive timing"]
end

subgraph group_evaluation["Online and Benchmark Workflows"]
  node_ovobench["OVO-Bench workflows"]
  node_streambench["StreamingBench workflows"]
  node_mmlife["MM-Lifelong workflows"]
  node_egolife["EgoLife construction"]
  node_holoassist["HoloAssist workflows"]
  node_cook["CaptainCook4D workflows"]
end

node_wearer(("Wearer"))
node_stream["Video stream"]

node_stream -->|"captions windows"| node_caption
node_caption -->|"supplies records"| node_builder
node_builder -->|"segments captions"| node_segment
node_builder -->|"adds episodes"| node_memorygraph
node_builder -->|"aggregates topics"| node_topic
node_memorygraph -->|"indexes memory"| node_dualindex
node_memorygraph -->|"retrieves from"| node_retriever
node_dualindex -->|"searches candidates"| node_retriever
node_retriever -.->|"provides recall"| node_skills
node_wearer -->|"asks about history"| node_qa
node_qa -->|"chooses retrieval"| node_skills
node_qa -.->|"requests evidence"| node_retriever
node_wearer -->|"presents situation"| node_services
node_services -.->|"retrieves evidence"| node_retriever
node_services -->|"reads shared memory"| node_memorygraph
node_wearer -.->|"streams experience"| node_estp
node_stream -->|"indexes visual content"| node_visual
node_stream -->|"evaluates online understanding"| node_ovobench
node_stream -->|"evaluates streaming understanding"| node_streambench
node_memorygraph -->|"supports long-horizon QA"| node_mmlife
node_stream -->|"builds stream memory"| node_egolife
node_stream -->|"supports service benchmark"| node_holoassist
node_stream -->|"supports service benchmark"| node_cook

click node_caption "https://github.com/sitonggong/grove/blob/main/hypervideo/modules/structured_perception.py"
click node_segment "https://github.com/sitonggong/grove/blob/main/hypervideo/extractors/episode_segmenter.py"
click node_builder "https://github.com/sitonggong/grove/blob/main/hypervideo/extractors/hypergraph_builder.py"
click node_topic "https://github.com/sitonggong/grove/blob/main/hypervideo/extractors/topic_aggregator.py"
click node_memorygraph "https://github.com/sitonggong/grove/blob/main/hypervideo/structure.py"
click node_dualindex "https://github.com/sitonggong/grove/blob/main/hypervideo/index/dual_index.py"
click node_retriever "https://github.com/sitonggong/grove/blob/main/hypervideo/retrieval/hierarchical_retriever.py"
click node_skills "https://github.com/sitonggong/grove/blob/main/hypervideo/skills/__init__.py"
click node_qa "https://github.com/sitonggong/grove/blob/main/hypervideo/sp_agentic_qa_v2.py"
click node_hierqa "https://github.com/sitonggong/grove/blob/main/hypervideo/sp_hierarchical.py"
click node_services "https://github.com/sitonggong/grove/blob/main/hypervideo/egoserve_service_engine_v4.py"
click node_estp "https://github.com/sitonggong/grove/blob/main/hypervideo/estp_streaming_inference.py"
click node_visual "https://github.com/sitonggong/grove/blob/main/hypervideo/visual_indexing.py"
click node_ovobench "https://github.com/sitonggong/grove/blob/main/hypervideo/ovobench_hyper_processing.py"
click node_streambench "https://github.com/sitonggong/grove/blob/main/hypervideo/streamingbench_hyper_processing.py"
click node_mmlife "https://github.com/sitonggong/grove/blob/main/hypervideo/mmlifelong_processing.py"
click node_egolife "https://github.com/sitonggong/grove/blob/main/hypervideo/egolife_hyper_processing.py"
click node_holoassist "https://github.com/sitonggong/grove/blob/main/hypervideo/holoassist_hyper_processing.py"
click node_cook "https://github.com/sitonggong/grove/blob/main/hypervideo/captaincook4d_hyper_processing.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_caption,node_segment,node_builder,node_topic toneBlue
class node_memorygraph,node_dualindex,node_retriever,node_skills,node_visual toneAmber
class node_qa,node_hierqa,node_services,node_estp toneMint
class node_ovobench,node_streambench,node_mmlife,node_egolife,node_holoassist,node_cook toneRose
class node_wearer,node_stream toneIndigo
```

## Components

### Memory Construction

- [Perceptual captioning](https://github.com/sitonggong/grove/blob/main/hypervideo/modules/structured_perception.py): `hypervideo/modules/structured_perception.py`
- [Episode segmentation](https://github.com/sitonggong/grove/blob/main/hypervideo/extractors/episode_segmenter.py): `hypervideo/extractors/episode_segmenter.py`
- [Hypergraph builder](https://github.com/sitonggong/grove/blob/main/hypervideo/extractors/hypergraph_builder.py): `hypervideo/extractors/hypergraph_builder.py`
- [Topic aggregation](https://github.com/sitonggong/grove/blob/main/hypervideo/extractors/topic_aggregator.py): `hypervideo/extractors/topic_aggregator.py`

### Memory and Retrieval

- [Stratified memory](https://github.com/sitonggong/grove/blob/main/hypervideo/structure.py): `hypervideo/structure.py`
- [Dual index](https://github.com/sitonggong/grove/blob/main/hypervideo/index/dual_index.py): `hypervideo/index/dual_index.py`
- [Hierarchical retrieval](https://github.com/sitonggong/grove/blob/main/hypervideo/retrieval/hierarchical_retriever.py): `hypervideo/retrieval/hierarchical_retriever.py`
- [Retrieval skills](https://github.com/sitonggong/grove/blob/main/hypervideo/skills/__init__.py): `hypervideo/skills/__init__.py`
- [Visual indexing](https://github.com/sitonggong/grove/blob/main/hypervideo/visual_indexing.py): `hypervideo/visual_indexing.py`

### Assistant Workflows

- [Reactive QA agent](https://github.com/sitonggong/grove/blob/main/hypervideo/sp_agentic_qa_v2.py): `hypervideo/sp_agentic_qa_v2.py`
- [Hierarchical QA](https://github.com/sitonggong/grove/blob/main/hypervideo/sp_hierarchical.py): `hypervideo/sp_hierarchical.py`
- [Proactive service](https://github.com/sitonggong/grove/blob/main/hypervideo/egoserve_service_engine_v4.py): `hypervideo/egoserve_service_engine_v4.py`
- [Proactive timing](https://github.com/sitonggong/grove/blob/main/hypervideo/estp_streaming_inference.py): `hypervideo/estp_streaming_inference.py`

### Online and Benchmark Workflows

- [OVO-Bench workflows](https://github.com/sitonggong/grove/blob/main/hypervideo/ovobench_hyper_processing.py): `hypervideo/ovobench_hyper_processing.py`
- [StreamingBench workflows](https://github.com/sitonggong/grove/blob/main/hypervideo/streamingbench_hyper_processing.py): `hypervideo/streamingbench_hyper_processing.py`
- [MM-Lifelong workflows](https://github.com/sitonggong/grove/blob/main/hypervideo/mmlifelong_processing.py): `hypervideo/mmlifelong_processing.py`
- [EgoLife construction](https://github.com/sitonggong/grove/blob/main/hypervideo/egolife_hyper_processing.py): `hypervideo/egolife_hyper_processing.py`
- [HoloAssist workflows](https://github.com/sitonggong/grove/blob/main/hypervideo/holoassist_hyper_processing.py): `hypervideo/holoassist_hyper_processing.py`
- [CaptainCook4D workflows](https://github.com/sitonggong/grove/blob/main/hypervideo/captaincook4d_hyper_processing.py): `hypervideo/captaincook4d_hyper_processing.py`

### Other components

- **Wearer**
- **Video stream**

## Connections

- Video stream → Perceptual captioning: captions windows
- Perceptual captioning → Hypergraph builder: supplies records
- Hypergraph builder → Episode segmentation: segments captions (evidence: `hypervideo/extractors/hypergraph_builder.py`)
- Hypergraph builder → Stratified memory: adds episodes (evidence: `hypervideo/extractors/hypergraph_builder.py`)
- Hypergraph builder → Topic aggregation: aggregates topics (evidence: `hypervideo/extractors/hypergraph_builder.py`)
- Stratified memory → Dual index: indexes memory (evidence: `hypervideo/index/dual_index.py`)
- Stratified memory → Hierarchical retrieval: retrieves from (evidence: `hypervideo/retrieval/hierarchical_retriever.py`)
- Dual index → Hierarchical retrieval: searches candidates (evidence: `hypervideo/retrieval/hierarchical_retriever.py`)
- Hierarchical retrieval → Retrieval skills: provides recall
- Wearer → Reactive QA agent: asks about history
- Reactive QA agent → Retrieval skills: chooses retrieval
- Reactive QA agent → Hierarchical retrieval: requests evidence
- Wearer → Proactive service: presents situation
- Proactive service → Hierarchical retrieval: retrieves evidence
- Proactive service → Stratified memory: reads shared memory
- Wearer → Proactive timing: streams experience
- Video stream → Visual indexing: indexes visual content (evidence: `hypervideo/visual_indexing.py`)
- Video stream → OVO-Bench workflows: evaluates online understanding
- Video stream → StreamingBench workflows: evaluates streaming understanding
- Stratified memory → MM-Lifelong workflows: supports long-horizon QA
- Video stream → EgoLife construction: builds stream memory
- Video stream → HoloAssist workflows: supports service benchmark
- Video stream → CaptainCook4D workflows: supports service benchmark
