How atunass/aic_2023 works
Project Type and Purpose
• This is a full-stack, multi-modal video retrieval system for “Event Retrieval from Video,” combining a user-facing web UI (Flask) with a RESTful search API (FastAPI), backed by computer-vision and NLP models (BLIP, FAISS, TransNetV2, InternVideo, YouTube transcript).
• Users can submit text, image, or ASR queries; the system retrieves matching news video segments by consulting precomputed feature stores and on-the-fly summarization pipelines.
Core Directory and Component Breakdown
• app_flask.py + templates/ + static/
• Flask web server, Jinja2 HTML templates (home.html, video.html, show_data.html), CSS/JS under static/team3 → presents UI and collects user queries.
• fastAPI_local.py
• FastAPI application exposing endpoints for text_search, image_search, asr_search, rerank, get_frame, show_frame, submit.
• configs/
• YAML/JSON config files defining model parameters for BLIP, VQA, retrieval tasks, etc.
• utils/
• Helper modules: feature extractors (My_BLIP.py, My_Faiss.py), video downloader (get_video.py), JSON format utilities, plus utils/models/ containing BLIP model wrappers.
• video_summary/InternVideo/
• InternVideo pipeline code for video summarization, tokenization, clip utilities—used for clip-level feature extraction and indexing.
• asr/
• JSON transcripts from ASR, used for ASR-based search.
• lst_scenes/ & lst_scenes_video/
• Precomputed lists of scene keyframe filenames, organized by level/batch; used when retrieving video frames.
• notebook/
• Jupyter notebooks for offline experimentation (Annoy vs. Faiss indexing, BLIP tests, InternVideo demo).
• requirements.txt
• Declares dependencies: Faiss (CPU/GPU), BLIP, Flask, FastAPI, InternVideo, youtube-transcript-api, etc.
High-Level Architectural Components
a. Client Layer (Browser)
• HTML/JS/CSS interface served by Flask; collects user input (text/image/ASR query) and displays results (thumbnails, video frames).
b. Web Server (Flask)
• Routes: “/” (home), “/search”, “/video”; serves static assets; forwards API calls to FastAPI or fetches precomputed data.
c. API Layer (FastAPI)
• Exposes micro-endpoints:
• /text_search → invokes utils/My_BLIP + My_Faiss + FAISS index lookup
• /image_search → extracts BLIP image features and queries FAISS
• /asr_search → scans ASR JSON transcripts
• /rerank → applies BLIP cross-encoder or BERT re-ranking
• /get_frame → fetches keyframe file paths from lst_scenes_*/
• /submit → collects final result for tournament submission
d. Computation/Model Services
• BLIP models (utils/models/blip*.py) for feature extraction / cross-modal scoring
• FAISS (utils/My_Faiss.py) for high-speed nearest-neighbor search over precomputed embeddings
• InternVideo (video_summary/InternVideo) for temporal summarization and additional feature generation
• TransNetV2 for shot boundary detection (if integrated)
• YouTube Transcript API for fetching ASR transcripts on demand
e. Data Storage Layer
• Feature Stores:
• blip_bin_json/ and interb_bin_json/ directories with .bin/.json feature files
• FAISS index files (persisted on disk)
• Scene Lists: lst_scenes/ & lst_scenes_video/ text files listing keyframe filenames by scene
• ASR JSON: asr/ folder
• Raw Video Files (not in repo but referenced by get_video.py)
Relationships and Data Flow
1. User issues query in browser → Flask endpoint
2. Flask either serves static UI or proxies REST call to FastAPI
3. FastAPI endpoint:
a. Loads appropriate model(s) via utils/models or video_summary modules
b. Extracts or looks up features (image embedding, text embedding, ASR text scan)
c. Queries FAISS for nearest neighbors (keyframe or shot IDs)
d. Optionally reranks with BLIP cross-encoder or BERT config
e. Gathers scene list entries and ASR/video frame paths
f. Returns JSON of ranked video IDs + keyframe paths + metadata
4. Flask receives JSON → populates Jinja template → user sees ranked thumbnails and can click through to video frames/video player
Key Architectural Patterns and Principles
• Separation of Concerns: UI (Flask) vs. search logic (FastAPI) vs. model code (utils/models & video_summary) vs. data store
• Microservice-like Endpoints: each search modality implemented as a separate FastAPI route
• Precompute and Cache: bulk feature extraction offline (bin/json, FAISS indexes, scene lists) to optimize online latency
• Config-Driven: all model and retrieval hyperparameters live in configs/ allowing easy swap of tasks (VQA, NLVR, retrieval).
Essential Elements
• Clear component names (Flask, FastAPI, BLIP, FAISS, InternVideo)
• Data repositories (bin/json directories, FAISS indexes, asr folder, lst_scenes)