# gabrielegoletto/amego architecture

> Architecture diagram of the GitHub repository gabrielegoletto/amego, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gabrielegoletto/amego
- Repository: https://github.com/gabrielegoletto/amego

## Overview

##### Main Components Identification

The system should be divided into three major processing pipelines:

a) Data Preprocessing Pipeline:
- Video Frame Extraction
- Optical Flow Generation (FlowFormer)
- Hand-Object Detection
- Data Storage Management

b) AMEGO Core Processing:
- Interaction Tracklets Extraction (HOI_AMEGO)
- Location Segments Extraction (LS_AMEGO)
- Feature Extraction (DINO)

c) Query System:
- AMB Query Processing (Q1-Q8)
- Result Generation

**Input Layer:**
**[Raw Egocentric Video Data]**
- ↓
**Preprocessing Layer:**
**[Frame Extraction] → [FlowFormer Flow] → [Hand-Object Detection]**
- ↓
**Storage Layer:**
**[RGB Frames] [Flow Data] [Hand-Object Detections]**
- ↓
**AMEGO Processing Layer:**
**[HOI Tracklet Extraction] ← → [Location Segment Extraction]**
- ↓
**Feature Processing:**
**[DINO Feature Extraction]**
- ↓
**Output Layer:**
**[Interaction Tracklets JSON] [Location Segments JSON]**
- ↓
**Query Interface:**
**[Q1-Q8 Query Processors]**

##### Component Details

- Rectangles: Processing modules
- Diamonds: Decision points

- Blue: Input/Output data
- Green: Core processing modules
- Yellow: Storage components
- Orange: Query system
- Gray: External dependencies

##### Important Elements

a) Data Formats:
- Video frames (456x256 resolution)
- Flow data (.pth files)
- Detection data (.pkl files)
- Output JSONs with specified fields

b) External Dependencies:
- FlowFormer
- Hand-Object Detector
- EgoTracks (STARK Tracker)
- DINO Feature Extractor

c) Configuration Management:
- Configuration flow from default.yaml
- CLI parameter override capability

##### Specific Implementation Details

Subcomponents for each major module:

**HOI Tracklet Extraction:**
- Track ID Management
- Object Bbox Processing
- Frame Sequence Tracking
- Feature Extraction
- Cluster Assignment

**Location Segment Extraction:**
- Cluster Management
- Feature Processing
- Frame Sequence Management

**Query System:**
- Query Type Selection (Q1-Q8)
- Data Retrieval
- Result Processing

##### Integration Points

Clear integration points between:
- Preprocessing pipeline outputs
- AMEGO core processing
- Query system inputs
- Storage system interaction

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Legend
        style Legend fill:#f5f5f5,stroke:#333
        L1[Processing Module]:::process
        L2[(Storage)]:::storage
        L3{Decision Point}:::decision
        L4([External Dependency]):::external
        L5[/API or Interface/]:::api
    end

    classDef process fill:#90EE90,stroke:#333
    classDef storage fill:#FFE4B5,stroke:#333
    classDef decision fill:#FFB6C1,stroke:#333
    classDef external fill:#B0C4DE,stroke:#333
    classDef api fill:#DDA0DD,stroke:#333
    classDef input fill:#87CEEB,stroke:#333

    Input[Raw Egocentric Video]:::input

    subgraph PreprocessingLayer
        FrameExtraction[Frame Extraction]:::process
        FlowGeneration[FlowFormer Flow Generation]:::process
        HandObjectDetection[Hand-Object Detection]:::process
        
        click FrameExtraction "https://github.com/gabrielegoletto/AMEGO/blob/main/submodules/EgoTracks/tools/preprocess/extract_ego4d_clip_frames.py"
        click FlowGeneration "https://github.com/gabrielegoletto/AMEGO/blob/main/tools/generate_flowformer_flow.py"
        click HandObjectDetection "https://github.com/gabrielegoletto/AMEGO/tree/main/submodules/hand_object_detector/"
    end

    subgraph StorageLayer
        RGBFrames[(RGB Frames)]:::storage
        FlowData[(Flow Data)]:::storage
        DetectionData[(Detection Data)]:::storage
    end

    subgraph ProcessingLayer
        HOITracklets[HOI Tracklet Extraction]:::process
        LocationSegments[Location Segment Extraction]:::process
        FeatureExtraction[DINO Feature Extraction]:::process
        
        click HOITracklets "https://github.com/gabrielegoletto/AMEGO/blob/main/HOI_AMEGO.py"
        click LocationSegments "https://github.com/gabrielegoletto/AMEGO/blob/main/LS_AMEGO.py"
    end

    subgraph ExternalDeps
        FlowFormer([FlowFormer]):::external
        ObjectTracker([STARK Tracker]):::external
        DINO([DINO Extractor]):::external
        
        click FlowFormer "https://github.com/gabrielegoletto/AMEGO/tree/main/submodules/flowformer/"
        click ObjectTracker "https://github.com/gabrielegoletto/AMEGO/tree/main/submodules/EgoTracks/tracking/models/stark_tracker/"
    end

    subgraph QuerySystem
        QueryProcessor[/Query Processor Q1-Q8/]:::api
        QueryResults[(Query Results)]:::storage
        
        click QueryProcessor "https://github.com/gabrielegoletto/AMEGO/tree/main/querying/"
    end

    subgraph ConfigSystem
        Config[Configuration Management]:::process
        
        click Config "https://github.com/gabrielegoletto/AMEGO/blob/main/configs/default.yaml"
    end

    Input --> FrameExtraction
    FrameExtraction --> RGBFrames
    RGBFrames --> FlowGeneration
    FlowGeneration --> FlowData
    RGBFrames --> HandObjectDetection
    HandObjectDetection --> DetectionData

    FlowFormer --> FlowGeneration
    ObjectTracker --> HOITracklets
    DINO --> FeatureExtraction

    RGBFrames --> HOITracklets
    FlowData --> HOITracklets
    DetectionData --> HOITracklets

    RGBFrames --> LocationSegments
    FlowData --> LocationSegments
    DetectionData --> LocationSegments

    HOITracklets --> FeatureExtraction
    LocationSegments --> FeatureExtraction

    FeatureExtraction --> QueryProcessor
    QueryProcessor --> QueryResults

    Config --> PreprocessingLayer
    Config --> ProcessingLayer
    Config --> QuerySystem
```
