How gabrielegoletto/amego works
Main Components Identification
The system should be divided into three major processing pipelines:
a) Data Preprocessing Pipeline:
• Video Frame Extraction
• Optical Flow Generation (FlowFormer)
• Hand-Object Detection
• Data Storage Management
b) AMEGO Core Processing:
• Interaction Tracklets Extraction (HOI_AMEGO)
• Location Segments Extraction (LS_AMEGO)
• Feature Extraction (DINO)
c) Query System:
• AMB Query Processing (Q1-Q8)
• Result Generation
Input Layer:
[Raw Egocentric Video Data]
• ↓
Preprocessing Layer:
[Frame Extraction] → [FlowFormer Flow] → [Hand-Object Detection]
• ↓
Storage Layer:
[RGB Frames] [Flow Data] [Hand-Object Detections]
• ↓
AMEGO Processing Layer:
[HOI Tracklet Extraction] ← → [Location Segment Extraction]
• ↓
Feature Processing:
[DINO Feature Extraction]
• ↓
Output Layer:
[Interaction Tracklets JSON] [Location Segments JSON]
• ↓
Query Interface:
[Q1-Q8 Query Processors]
Component Details
• Rectangles: Processing modules
• Diamonds: Decision points
• Blue: Input/Output data
• Green: Core processing modules
• Yellow: Storage components
• Orange: Query system
• Gray: External dependencies
Important Elements
a) Data Formats:
• Video frames (456x256 resolution)
• Flow data (.pth files)
• Detection data (.pkl files)
• Output JSONs with specified fields
b) External Dependencies:
• FlowFormer
• Hand-Object Detector
• EgoTracks (STARK Tracker)
• DINO Feature Extractor
c) Configuration Management:
• Configuration flow from default.yaml
• CLI parameter override capability
Specific Implementation Details
Subcomponents for each major module:
HOI Tracklet Extraction:
• Track ID Management
• Object Bbox Processing
• Frame Sequence Tracking
• Feature Extraction
• Cluster Assignment
Location Segment Extraction:
• Cluster Management
• Feature Processing
• Frame Sequence Management
Query System:
• Query Type Selection (Q1-Q8)
• Data Retrieval
• Result Processing
Integration Points
Clear integration points between:
• Preprocessing pipeline outputs
• AMEGO core processing
• Query system inputs
• Storage system interaction