How sitonggong/grove works
GROVE is a training-free streaming-video memory for both reactive history QA and proactive assistance. Its documented construction flow captions incoming video, segments activity into episodes, extracts facts and aggregates recurring topics, then keeps the resulting memory and indexes queryable. Reactive agents choose retrieval skills over multiple rounds; proactive agents use the current situation to assemble evidence. The tree also contains benchmark-specific processing and inference entry points. Relationships among those unsampled entry points are not expanded; edges below rely on README statements or shown source excerpts.
Main entry points: structure.py Stratified memory · hierarchical_retriever.py Hierarchical retrieval · hypergraph_builder.py Hypergraph builder · sp_agentic_qa_v2.py Reactive QA agent
Components 21
Memory Construction
- Perceptual captioning
hypervideo/modules/structured_perception.py - Episode segmentation
hypervideo/extractors/episode_segmenter.py - Hypergraph builder
hypervideo/extractors/hypergraph_builder.py - Topic aggregation
hypervideo/extractors/topic_aggregator.py
Memory and Retrieval
- Stratified memory
hypervideo/structure.py - Dual index
hypervideo/index/dual_index.py - Hierarchical retrieval
hypervideo/retrieval/hierarchical_retriever.py - Retrieval skills
hypervideo/skills/__init__.py - Visual indexing
hypervideo/visual_indexing.py
Assistant Workflows
- Reactive QA agent
hypervideo/sp_agentic_qa_v2.py - Hierarchical QA
hypervideo/sp_hierarchical.py - Proactive service
hypervideo/egoserve_service_engine_v4.py - Proactive timing
hypervideo/estp_streaming_inference.py
Online and Benchmark Workflows
- OVO-Bench workflows
hypervideo/ovobench_hyper_processing.py - StreamingBench workflows
hypervideo/streamingbench_hyper_processing.py - MM-Lifelong workflows
hypervideo/mmlifelong_processing.py - EgoLife construction
hypervideo/egolife_hyper_processing.py - HoloAssist workflows
hypervideo/holoassist_hyper_processing.py - CaptainCook4D workflows
hypervideo/captaincook4d_hyper_processing.py
Other components
- Wearer
- Video stream
Connections 23
- Video stream → Perceptual captioning: captions windows
- Perceptual captioning → Hypergraph builder: supplies records
- Hypergraph builder → Episode segmentation: segments captions
hypervideo/extractors/hypergraph_builder.py - Hypergraph builder → Stratified memory: adds episodes
hypervideo/extractors/hypergraph_builder.py - Hypergraph builder → Topic aggregation: aggregates topics
hypervideo/extractors/hypergraph_builder.py - Stratified memory → Dual index: indexes memory
hypervideo/index/dual_index.py - Stratified memory → Hierarchical retrieval: retrieves from
hypervideo/retrieval/hierarchical_retriever.py - Dual index → Hierarchical retrieval: searches candidates
hypervideo/retrieval/hierarchical_retriever.py - Hierarchical retrieval → Retrieval skills: provides recall
- Wearer → Reactive QA agent: asks about history
- Reactive QA agent → Retrieval skills: chooses retrieval
- Reactive QA agent → Hierarchical retrieval: requests evidence
- Wearer → Proactive service: presents situation
- Proactive service → Hierarchical retrieval: retrieves evidence
- Proactive service → Stratified memory: reads shared memory
- Wearer → Proactive timing: streams experience
- Video stream → Visual indexing: indexes visual content
hypervideo/visual_indexing.py - Video stream → OVO-Bench workflows: evaluates online understanding
- Video stream → StreamingBench workflows: evaluates streaming understanding
- Stratified memory → MM-Lifelong workflows: supports long-horizon QA
- Video stream → EgoLife construction: builds stream memory
- Video stream → HoloAssist workflows: supports service benchmark
- Video stream → CaptainCook4D workflows: supports service benchmark