# atunass/aic_2023 architecture

> Architecture diagram of the GitHub repository atunass/aic_2023, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/atunass/aic_2023
- Repository: https://github.com/atunass/aic_2023

## Overview

##### Project Type and Purpose
- This is a full-stack, multi-modal video retrieval system for “Event Retrieval from Video,” combining a user-facing web UI (Flask) with a RESTful search API (FastAPI), backed by computer-vision and NLP models (BLIP, FAISS, TransNetV2, InternVideo, YouTube transcript).
- Users can submit text, image, or ASR queries; the system retrieves matching news video segments by consulting precomputed feature stores and on-the-fly summarization pipelines.

##### Core Directory and Component Breakdown
- **app_flask.py + templates/ + static/**
- Flask web server, Jinja2 HTML templates (home.html, video.html, show_data.html), CSS/JS under static/team3 → presents UI and collects user queries.
- **fastAPI_local.py**
- FastAPI application exposing endpoints for text_search, image_search, asr_search, rerank, get_frame, show_frame, submit.
- **configs/**
- YAML/JSON config files defining model parameters for BLIP, VQA, retrieval tasks, etc.
- **utils/**
- Helper modules: feature extractors (My_BLIP.py, My_Faiss.py), video downloader (get_video.py), JSON format utilities, plus `utils/models/` containing BLIP model wrappers.
- **video_summary/InternVideo/**
- InternVideo pipeline code for video summarization, tokenization, clip utilities—used for clip-level feature extraction and indexing.
- **asr/**
- JSON transcripts from ASR, used for ASR-based search.
- **lst_scenes/ & lst_scenes_video/**
- Precomputed lists of scene keyframe filenames, organized by level/batch; used when retrieving video frames.
- **notebook/**
- Jupyter notebooks for offline experimentation (Annoy vs. Faiss indexing, BLIP tests, InternVideo demo).
- **requirements.txt**
- Declares dependencies: Faiss (CPU/GPU), BLIP, Flask, FastAPI, InternVideo, youtube-transcript-api, etc.

##### High-Level Architectural Components
a. **Client Layer (Browser)**
- HTML/JS/CSS interface served by Flask; collects user input (text/image/ASR query) and displays results (thumbnails, video frames).
b. **Web Server (Flask)**
- Routes: “/” (home), “/search”, “/video”; serves static assets; forwards API calls to FastAPI or fetches precomputed data.
c. **API Layer (FastAPI)**
- Exposes micro-endpoints:
- /text_search → invokes utils/My_BLIP + My_Faiss + FAISS index lookup
- /image_search → extracts BLIP image features and queries FAISS
- /asr_search → scans ASR JSON transcripts
- /rerank → applies BLIP cross-encoder or BERT re-ranking
- /get_frame → fetches keyframe file paths from lst_scenes_*/
- /submit → collects final result for tournament submission
d. **Computation/Model Services**
- **BLIP models** (utils/models/blip*.py) for feature extraction / cross-modal scoring
- **FAISS** (utils/My_Faiss.py) for high-speed nearest-neighbor search over precomputed embeddings
- **InternVideo** (video_summary/InternVideo) for temporal summarization and additional feature generation
- **TransNetV2** for shot boundary detection (if integrated)
- **YouTube Transcript API** for fetching ASR transcripts on demand
e. **Data Storage Layer**
- **Feature Stores**:
- `blip_bin_json/` and `interb_bin_json/` directories with .bin/.json feature files
- FAISS index files (persisted on disk)
- **Scene Lists**: `lst_scenes/` & `lst_scenes_video/` text files listing keyframe filenames by scene
- **ASR JSON**: `asr/` folder
- **Raw Video Files** (not in repo but referenced by get_video.py)

##### Relationships and Data Flow
1. User issues query in browser → Flask endpoint
2. Flask either serves static UI or proxies REST call to FastAPI
3. FastAPI endpoint:
a. Loads appropriate model(s) via utils/models or video_summary modules
b. Extracts or looks up features (image embedding, text embedding, ASR text scan)
c. Queries FAISS for nearest neighbors (keyframe or shot IDs)
d. Optionally reranks with BLIP cross-encoder or BERT config
e. Gathers scene list entries and ASR/video frame paths
f. Returns JSON of ranked video IDs + keyframe paths + metadata
4. Flask receives JSON → populates Jinja template → user sees ranked thumbnails and can click through to video frames/video player

##### Key Architectural Patterns and Principles
- **Separation of Concerns**: UI (Flask) vs. search logic (FastAPI) vs. model code (utils/models & video_summary) vs. data store
- **Microservice-like Endpoints**: each search modality implemented as a separate FastAPI route
- **Precompute and Cache**: bulk feature extraction offline (bin/json, FAISS indexes, scene lists) to optimize online latency
- **Config-Driven**: all model and retrieval hyperparameters live in `configs/` allowing easy swap of tasks (VQA, NLVR, retrieval).

##### Essential Elements
- Clear component names (Flask, FastAPI, BLIP, FAISS, InternVideo)
- Data repositories (bin/json directories, FAISS indexes, asr folder, lst_scenes)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Frontend Layer
    subgraph "Client Layer" 
        direction TB
        Browser["Browser (HTML/CSS/JS)"]:::frontend
        Flask["Flask Web Server"]:::frontend
        Templates["Flask Templates"]:::frontend
        Static["Static Assets"]:::frontend
    end

    %% API Layer
    subgraph "API Layer"
        direction TB
        FastAPI["FastAPI Search Service"]:::api
    end

    %% Compute/Model Tier
    subgraph "Compute/Model Tier"
        direction TB
        BLIPUtil["BLIP Feature Extractor"]:::compute
        FAISSUtil["FAISS Search Utility"]:::compute
        VideoDL["Video Downloader Utility"]:::compute
        JSONFmt["JSON Path Formatter Utility"]:::compute
        ASRTest["ASR Transcript Utilities & Tests"]:::compute
        subgraph "Model Wrappers"
            direction TB
            WrapperBLIP["BLIP Wrappers"]:::compute
            WrapperMED["MED, VQA, NLVR"]:::compute
            WrapperViT["ViT Wrapper"]:::compute
        end
        subgraph "InternVideo Pipeline"
            direction TB
            InternMain["InternVideo Core"]:::compute
            ClipUtils["InternVideo Clip Utils"]:::compute
            VideoDemo["Video Summary Demo"]:::compute
            VideoJSONFmt["Video Summary JSON Formatter"]:::compute
        end
    end

    %% Data Layer
    subgraph "Data Storage Layer"
        direction TB
        Configs["Configuration Files"]:::data
        ASRData["ASR JSON Transcripts"]:::data
        SceneText["Scene Keyframe Lists (lst_scenes)"]:::data
        SceneVideo["Scene Keyframe Lists Video (lst_scenes_video)"]:::data
        Requirements["Requirements.txt"]:::data
        Notebooks["Jupyter Notebooks"]:::data
        BLIPTests["BLIP Test Script"]:::data
        InternTests["InternVideo Test Script"]:::data
    end

    %% External Services
    subgraph "External Services"
        direction TB
        YouTubeAPI["YouTube Transcript API"]:::external
    end

    %% Connections
    Browser -->|"UI Requests"| Flask
    Flask -->|"Proxies REST Calls"| FastAPI
    FastAPI -->|"uses"| BLIPUtil
    FastAPI -->|"uses"| FAISSUtil
    FastAPI -->|"calls"| VideoDL
    FastAPI -->|"formats JSON"| JSONFmt
    FastAPI -->|"calls ASR Tests"| ASRTest
    FastAPI -->|"invokes"| WrapperBLIP
    FastAPI -->|"invokes"| WrapperMED
    FastAPI -->|"invokes"| WrapperViT
    FastAPI -->|"uses"| InternMain
    InternMain -->|"uses"| ClipUtils
    InternMain -->|"formats JSON"| VideoJSONFmt
    InternMain -->|"demo via"| VideoDemo
    FastAPI -->|"fetches transcripts"| YouTubeAPI
    BLIPUtil -->|"loads configs"| Configs
    FAISSUtil -->|"loads indexes"| Configs
    ASRTest -->|"reads transcripts"| ASRData
    FastAPI -->|"reads"| ASRData
    FastAPI -->|"reads"| SceneText
    FastAPI -->|"reads"| SceneVideo
    Flask -->|"serves templates"| Templates
    Flask -->|"serves static"| Static
    Notebooks -->|"experiments with"| BLIPUtil
    Notebooks -->|"experiments with"| FAISSUtil
    Notebooks -->|"experiments with"| InternMain
    BLIPTests -->|"tests"| BLIPUtil
    InternTests -->|"tests"| InternMain

    %% Click Events
    click Flask "https://github.com/atunass/aic_2023/blob/main/app_flask.py"
    click Templates "https://github.com/atunass/aic_2023/tree/main/templates/"
    click Static "https://github.com/atunass/aic_2023/blob/main/static/team3/css/my_styles.css"
    click Static "https://github.com/atunass/aic_2023/blob/main/static/team3/js/my_script.js"
    click FastAPI "https://github.com/atunass/aic_2023/blob/main/fastAPI_local.py"
    click Configs "https://github.com/atunass/aic_2023/tree/main/configs/"
    click BLIPUtil "https://github.com/atunass/aic_2023/blob/main/utils/My_BLIP.py"
    click FAISSUtil "https://github.com/atunass/aic_2023/blob/main/utils/My_Faiss.py"
    click VideoDL "https://github.com/atunass/aic_2023/blob/main/utils/get_video.py"
    click JSONFmt "https://github.com/atunass/aic_2023/blob/main/utils/format_json_path.py"
    click ASRTest "https://github.com/atunass/aic_2023/blob/main/utils/asr_test.py"
    click ASRData "https://github.com/atunass/aic_2023/tree/main/asr/"
    click WrapperBLIP "https://github.com/atunass/aic_2023/blob/main/utils/models/blip.py"
    click WrapperMED "https://github.com/atunass/aic_2023/blob/main/utils/models/med.py"
    click WrapperViT "https://github.com/atunass/aic_2023/blob/main/utils/models/vit.py"
    click InternMain "https://github.com/atunass/aic_2023/blob/main/video_summary/InternVideo/internvideo.py"
    click ClipUtils "https://github.com/atunass/aic_2023/tree/main/video_summary/InternVideo/clip_utils/"
    click VideoDemo "https://github.com/atunass/aic_2023/blob/main/video_summary/demo.py"
    click VideoJSONFmt "https://github.com/atunass/aic_2023/blob/main/video_summary/format_json_paths.py"
    click SceneText "https://github.com/atunass/aic_2023/tree/main/lst_scenes/"
    click SceneVideo "https://github.com/atunass/aic_2023/tree/main/lst_scenes_video/"
    click Notebooks "https://github.com/atunass/aic_2023/tree/main/notebook/"
    click Requirements "https://github.com/atunass/aic_2023/blob/main/requirements.txt"
    click BLIPTests "https://github.com/atunass/aic_2023/blob/main/blip_tests.py"
    click InternTests "https://github.com/atunass/aic_2023/blob/main/intern_test.py"

    %% Styles
    classDef frontend fill:#AEDFF7,stroke:#1E90FF,color:#000
    classDef api fill:#C8F7C5,stroke:#28A745,color:#000
    classDef compute fill:#FFD8A8,stroke:#FF8C00,color:#000
    classDef data fill:#E0E0E0,stroke:#808080,color:#000
    classDef external fill:#FFF5E1,stroke-dasharray: 5 5,stroke:#FFA500,color:#000
```
