How fpv-iplab/easg works
Project Type and Purpose
• This is a research codebase for:
– Preparing and annotating egocentric video data (Ego-EASG dataset)
– Running an AWS SageMaker–powered human-in-the-loop labeling system
– Training/inferencing a graph‐generation baseline (using object detectors and transformers)
– Performing downstream analyses on generated graphs
Top-Level Architecture Components
A. Data Sources & Storage
• Ego4D raw videos & original annotations (external storage)
• Processed frames & manifest files (S3 buckets / local FS)
• Downloaded Ego-EASG dataset zip
B. Input Preparation Pipeline
• Jupyter notebooks (input-preparation/*.ipynb) that:
– Convert raw annotations to JSON
– Extract PRE, PNR, POST frames
– Produce mapping files & SageMaker job lists
C. Labeling System (AWS SageMaker)
• SageMaker Custom Labeling Job
• Pre-condition Lambda functions (PRE-EASG)
• Consolidation Lambda functions (ACS-EASG)
• Second-stage validation Lambdas (easg-validation)
• Custom web UI templates (index.liquid.html) and static assets (JS/CSS under web/static) served from S3
D. Generation Baseline
• Data loader modules (easg-generation/dataloader)
• Faster R-CNN detection code (easg-generation/fasterRCNN)
• Core training & inference scripts (run_easg.py, train_with_EASG.py)
• Supporting libs (easg-generation/lib/…): optimizers, transformers, evaluation, utilities
• Config files and requirements
E. Downstream Analysis
• Jupyter notebook and JSON configs in easg-downstream (e.g., ALL_GRAPHS_FOR_DOWNSTREAM.json)
• Prompts for OpenAI API experiments
Relationships & Data Flows
1. Raw data → Input-Preparation notebooks → Frame extraction & JSON manifests → S3
2. Manifests + frame S3 paths → SageMaker Labeling Job
• Workers interact via web UI (JS front end)
• Pre-condition Lambda creates HITs
• Consolidation Lambda collects results → output labels stored back in S3
• Validation stage with similar flow under easg-validation
3. Labeled data + original annotations → Graph Generation
• Python scripts read frames & labels via data loader → object detection (Faster R-CNN) → feature vectors → transformer‐based graph generator → output scene graphs
4. Generated graphs → Downstream notebooks for evaluation / API experiments
Architectural Patterns & Design Principles
• Modular pipelines: separation of concerns between data prep, labeling, generation, analysis
• Serverless microservices: AWS Lambdas for HIT creation & consolidation
• Managed ML services: AWS SageMaker for scalable human labeling
• Pluggable detection & model code: clear data loader / model / utils layers
• Notebook-driven ETL & analysis
Final Checklist for Detail
• Both notebook ETL steps and production code flows
• Iterative loops (e.g., multiple labeling stages)
• Technology stacks: PyTorch/Faster R-CNN, AWS Lambda/SageMaker, Jupyter, OpenAI API