# fpv-iplab/easg architecture

> Architecture diagram of the GitHub repository fpv-iplab/easg, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fpv-iplab/easg
- Repository: https://github.com/fpv-iplab/easg

## Overview

##### Project Type and Purpose
- This is a research codebase for:
– Preparing and annotating egocentric video data (Ego-EASG dataset)
– Running an AWS SageMaker–powered human-in-the-loop labeling system
– Training/inferencing a graph‐generation baseline (using object detectors and transformers)
– Performing downstream analyses on generated graphs

##### Top-Level Architecture Components
A. Data Sources & Storage
- Ego4D raw videos & original annotations (external storage)
- Processed frames & manifest files (S3 buckets / local FS)
- Downloaded Ego-EASG dataset zip
B. Input Preparation Pipeline
- Jupyter notebooks (`input-preparation/*.ipynb`) that:
– Convert raw annotations to JSON
– Extract PRE, PNR, POST frames
– Produce mapping files & SageMaker job lists
C. Labeling System (AWS SageMaker)
- SageMaker Custom Labeling Job
- Pre-condition Lambda functions (PRE-EASG)
- Consolidation Lambda functions (ACS-EASG)
- Second-stage validation Lambdas (easg-validation)
- Custom web UI templates (`index.liquid.html`) and static assets (JS/CSS under `web/static`) served from S3
D. Generation Baseline
- Data loader modules (`easg-generation/dataloader`)
- Faster R-CNN detection code (`easg-generation/fasterRCNN`)
- Core training & inference scripts (`run_easg.py`, `train_with_EASG.py`)
- Supporting libs (`easg-generation/lib/…`): optimizers, transformers, evaluation, utilities
- Config files and requirements
E. Downstream Analysis
- Jupyter notebook and JSON configs in `easg-downstream` (e.g., ALL_GRAPHS_FOR_DOWNSTREAM.json)
- Prompts for OpenAI API experiments

##### Relationships & Data Flows
1. Raw data → Input-Preparation notebooks → Frame extraction & JSON manifests → S3
2. Manifests + frame S3 paths → SageMaker Labeling Job
- Workers interact via web UI (JS front end)
- Pre-condition Lambda creates HITs
- Consolidation Lambda collects results → output labels stored back in S3
- Validation stage with similar flow under `easg-validation`
3. Labeled data + original annotations → Graph Generation
- Python scripts read frames & labels via data loader → object detection (Faster R-CNN) → feature vectors → transformer‐based graph generator → output scene graphs
4. Generated graphs → Downstream notebooks for evaluation / API experiments

##### Architectural Patterns & Design Principles
- Modular pipelines: separation of concerns between data prep, labeling, generation, analysis
- Serverless microservices: AWS Lambdas for HIT creation & consolidation
- Managed ML services: AWS SageMaker for scalable human labeling
- Pluggable detection & model code: clear data loader / model / utils layers
- Notebook-driven ETL & analysis

##### Final Checklist for Detail
- Both notebook ETL steps and production code flows
- Iterative loops (e.g., multiple labeling stages)
- Technology stacks: PyTorch/Faster R-CNN, AWS Lambda/SageMaker, Jupyter, OpenAI API

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input Preparation & Labeling Zone
    subgraph "Input Preparation & Labeling"
        direction TB
        Ego4D["Ego4D Raw Videos & Annotations"]:::external
        EgoEASGzip["Ego-EASG Dataset Zip"]:::external
        Notebooks["Jupyter Notebooks\n(input-preparation/*.ipynb)"]:::compute
        MappingFiles["Mapping JSON Files\n(input-preparation/*.json)"]:::storage
        Manifests["JSON Manifests & Frames"]:::storage
        LocalFS["S3 Buckets / Local FS"]:::storage
    end

    %% AWS Labeling System
    subgraph "AWS Cloud"
        direction TB
        SageMakerJob["SageMaker Labeling Job"]:::compute
        subgraph "Labeling Lambdas"
            direction TB
            LambdaPre["PRE-EASG Lambda"]:::compute
            LambdaACS["ACS-EASG Lambda"]:::compute
        end
        subgraph "Validation Stage"
            direction TB
            LambdaPREVAL["PRE-EASG Validation Lambda"]:::compute
            LambdaACSVAL["ACS-EASG Validation Lambda"]:::compute
            ValidationUI["Validation Web Template\n(index.liquid.html)"]:::ui
        end
        UI["Web UI Template\n(index.html)"]:::ui
        StaticAssets["Static Assets\n(web/static)"]:::ui
    end

    %% Generation Baseline Zone
    subgraph "Generation Baseline"
        direction TB
        DataLoader["Data Loader Modules\n(easg-generation/dataloader)"]:::compute
        FRCNN["Faster R-CNN Module"]:::compute
        Training["Training & Inference\n(run_easg.py, train_with_EASG.py)"]:::compute
        Libs["Supporting Libs\n(easg-generation/lib)"]:::compute
        Utils["Utility Scripts\n(easg-generation/utils)"]:::compute
        Config["GETTING_STARTED.md"]:::storage
    end

    %% Downstream Analysis Zone
    subgraph "Downstream Analysis"
        direction TB
        DSNotebook["Analysis Notebook\n(EASG_action_anticipation_OpenAI_API.ipynb)"]:::compute
        DSConfigs["JSON Configs\n(ALL_GRAPHS_FOR_DOWNSTREAM.json,\nprompts.json)"]:::storage
    end

    %% Data Flows
    Ego4D -->|raw videos| Notebooks
    EgoEASGzip -->|downloaded zip| Notebooks
    Notebooks -->|produce mappings| MappingFiles
    MappingFiles -->|generate manifests & frames| Manifests
    Manifests -->|store frames & manifests| LocalFS
    LocalFS -->|input to labeling| SageMakerJob
    SageMakerJob -->|HIT UI| UI
    SageMakerJob -->|invoke| LambdaPre
    SageMakerJob -->|invoke| LambdaACS
    LambdaPre -->|pre-labels| LocalFS
    LambdaACS -->|consolidated labels| LocalFS
    LocalFS -->|validation inputs| LambdaPREVAL
    LocalFS -->|validation inputs| LambdaACSVAL
    LambdaPREVAL -->|validation results| LocalFS
    LambdaACSVAL -->|validation results| LocalFS
    LocalFS -->|labeled data| DataLoader
    DataLoader -->|load frames & labels| FRCNN
    FRCNN -->|features| Training
    Training -->|model code & utils| Libs
    Training -->|utility scripts| Utils
    Config -->|docs| Training
    Training -->|scene graphs| DSNotebook
    DSConfigs -->|config| DSNotebook

    %% Click Events
    click Notebooks "https://github.com/fpv-iplab/easg/tree/main/easg-labeling-system/input-preparation/"
    click MappingFiles "https://github.com/fpv-iplab/easg/tree/main/easg-labeling-system/input-preparation/"
    click LambdaPre "https://github.com/fpv-iplab/easg/blob/main/easg-labeling-system/easg-annotation/PRE-EASG/lambda_function.py"
    click LambdaACS "https://github.com/fpv-iplab/easg/tree/main/easg-labeling-system/easg-annotation/ACS-EASG/"
    click UI "https://github.com/fpv-iplab/easg/blob/main/easg-labeling-system/easg-annotation/index.html"
    click StaticAssets "https://github.com/fpv-iplab/easg/tree/main/easg-labeling-system/web/static"
    click LambdaPREVAL "https://github.com/fpv-iplab/easg/blob/main/easg-labeling-system/easg-validation/PRE-EASG-VALIDATION/lambda_function.py"
    click LambdaACSVAL "https://github.com/fpv-iplab/easg/tree/main/easg-labeling-system/easg-validation/ACS-EASG-VALIDATION/"
    click ValidationUI "https://github.com/fpv-iplab/easg/blob/main/easg-labeling-system/easg-validation/index.liquid.html"
    click DataLoader "https://github.com/fpv-iplab/easg/tree/main/easg-generation/dataloader/"
    click FRCNN "https://github.com/fpv-iplab/easg/tree/main/easg-generation/fasterRCNN/"
    click Libs "https://github.com/fpv-iplab/easg/tree/main/easg-generation/lib/"
    click Utils "https://github.com/fpv-iplab/easg/tree/main/easg-generation/utils/"
    click Config "https://github.com/fpv-iplab/easg/blob/main/easg-generation/GETTING_STARTED.md"
    click DSNotebook "https://github.com/fpv-iplab/easg/blob/main/easg-downstream/EASG_action_anticipation_OpenAI_API.ipynb"
    click DSConfigs "https://github.com/fpv-iplab/easg/tree/main/easg-downstream/"

    %% Styles
    classDef storage fill:#ADD8E6,stroke:#000;
    classDef compute fill:#90EE90,stroke:#000;
    classDef ui fill:#FFA500,stroke:#000;
    classDef external fill:#D3D3D3,stroke:#000;
```
