# xndien2004/llm_powered_video_search architecture

> Architecture diagram of the GitHub repository xndien2004/llm_powered_video_search, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xndien2004/llm_powered_video_search
- Repository: https://github.com/xndien2004/llm_powered_video_search

## Overview

##### Project type and purpose
- This is a full-stack web application: a multimedia/video retrieval system built with Django for the back end and plain HTML/CSS/JavaScript for the front end.
- Its purpose is to provide intelligent, multimodal video search (text, image, metadata) by combining offline data-extraction pipelines with online LLM-augmented retrieval.

##### File structure
- Top level:
- AIC/             ← Django project (settings, URLs, WSGI/ASGI)
- app/             ← Django “app” containing models, views, API endpoints, templates, static assets
- data_extraction/ ← offline scripts & notebooks for extracting frames, audio, metadata, TF-IDF encodings
- utils/           ← core retrieval logic: LLM retrieval, Faiss search, CLIP integration, audio/ocr/object tagging
- docker-compose.yml, manage.py, requirements.txt
- Architectural clues:
- app/viewAPI.py exposes REST endpoints that orchestrate calls into utils/
- data_extraction/* is an ETL pipeline run offline to populate indices and metadata stores
- Django ORM models in app/models.py back a relational database for metadata

##### README for additional insights
- Confirms technologies: Django, CLIP, Faiss, TF-IDF, PyTorch, OpenCV, Transformers

##### Main components
a. Data Extraction Pipeline (offline batch process)
- Raw Videos (source storage)
- TransnetV2 frame extraction
- Audio extraction & ASR
- TF-IDF encoding
→ Outputs: Feature files (NumPy/PyTorch tensors), metadata CSV/DB, Faiss indices
b. Backend (online service)
- Django REST API
- viewAPI.py orchestrator
- app/models.py → Relational DB (PostgreSQL/SQLite)
- Index store (Faiss on disk or in-memory)
c. Retrieval Engines (utils modules)
- CLIP-based image embedding + Faiss search
- TF-IDF text search
- Audio/ASR search
- OCR search
- LLM retriever (GPT-4 or local LLM via utils/LLM)
- combine_search merges scores
d. Frontend (client)
- HTML templates (index.html, video.html)
- JavaScript controllers (jquery_collect.js, others.js)
- CSS styling
e. External Services
- LLM API (OpenAI or Hugging Face)
- Docker for containerization (docker-compose)

##### Relationships and interactions
- Offline: Raw Videos → data_extraction pipeline → Feature store + Metadata DB + Faiss indices
- Online: Browser → Django URLs → viewAPI endpoint
→ calls retrieval engines in utils/
→ queries Metadata DB + Faiss indices + LLM API
→ aggregates & filters results
- Results returned as JSON → Frontend JS renders keyframes/video segments

##### Architectural patterns and design principles
- MVC pattern in Django (models/views/templates)
- Separation of offline ETL (data_extraction) vs. online retrieval (app + utils)
- Dependency injection: viewAPI imports modular utils search functions
- Indexing layer (Faiss) decoupled from relational metadata
- LLM integration as an external micro-service

##### Elements
- From raw video → extraction → indices
- From browser → Django → utils → DB/index → LLM → back to browser
- Green: Offline/batch processes
- Blue: Backend services
- Orange: Storage layers
- Yellow: External services
- Rectangles for services/modules

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Offline Pipeline
    subgraph "Offline Data Extraction Pipeline" 
        direction TB
        RawVideos["Raw Videos"]:::offline
        Transnet["TransnetV2 Frame Extraction"]:::offline
        AudioExt["Audio Extraction"]:::offline
        AudioDet["Audio Detection"]:::offline
        AudioRec["Audio Recognition (ASR)"]:::offline
        MetadataNB["Metadata Notebooks"]:::offline
        ImgFeat["Image Feature Extraction"]:::offline
        TFIDF["TF-IDF Encoding"]:::offline
        CreateBin["Binary/Feature File Creation"]:::offline
        RawVideos --> Transnet --> ImgFeat --> CreateBin
        RawVideos --> AudioExt --> AudioDet --> AudioRec --> CreateBin
        RawVideos --> MetadataNB --> CreateBin
        CreateBin -->|"features, tensors"| FaissIndex
        CreateBin -->|"metadata CSV/DB"| MetadataDB
    end

    %% Backend Service
    subgraph "Django Backend Service" 
        direction TB
        DjangoREST["Django REST API (viewAPI.py)"]:::backend
        URLRouting["URL Routing & Views"]:::backend
        ORMModels["ORM Models (app/models.py)"]:::backend
        Config["Project Config & URLs"]:::backend
        DjangoREST --> ORMModels
        URLRouting --> DjangoREST
        Config --> URLRouting
    end

    %% Retrieval Engines
    subgraph "Retrieval Engines (utils)" 
        direction TB
        FaissSearch["Faiss-based Vector Search"]:::retrieval
        TFIDFText["TF-IDF Text Search"]:::retrieval
        AudioSearch["Audio/ASR Search"]:::retrieval
        OCRSearch["OCR Search"]:::retrieval
        TagColor["Tag & Color Search"]:::retrieval
        LLMModule["LLM Retriever"]:::retrieval
        CombineSearch["Combine Search Scores"]:::retrieval
        Langchain["LangChain Adapters"]:::retrieval
        NLPProc["NLP Preprocessing"]:::retrieval
        VideoFuncs["Video-specific Retrieval"]:::retrieval
        DjangoREST --> FaissSearch
        DjangoREST --> TFIDFText
        DjangoREST --> AudioSearch
        DjangoREST --> OCRSearch
        DjangoREST --> TagColor
        DjangoREST --> LLMModule
        DjangoREST --> CombineSearch
        FaissSearch -->|"vectors"| FaissIndex
        TFIDFText --> MetadataDB
        AudioSearch --> MetadataDB
        OCRSearch --> MetadataDB
        TagColor --> MetadataDB
        LLMModule --> ExternalLLM
        CombineSearch --> DjangoREST
    end

    %% Frontend Client
    subgraph "Frontend Client" 
        direction TB
        HTMLIndex["index.html"]:::frontend
        HTMLVideo["video.html"]:::frontend
        JS1["jquery_colect.js"]:::frontend
        JS2["others.js"]:::frontend
        CSS1["index.css"]:::frontend
        CSS2["main.css"]:::frontend
        CSS3["chatbot.css"]:::frontend
        CSS4["header.css"]:::frontend
        CSS5["navLeft.css"]:::frontend
        CSS6["navRight.css"]:::frontend
        Browser["Browser"]:::frontend
        Browser -->|JSON request| DjangoREST
        HTMLIndex --> Browser
        HTMLVideo --> Browser
        JS1 --> Browser
        JS2 --> Browser
        CSS1 --> Browser
        CSS2 --> Browser
        CSS3 --> Browser
        CSS4 --> Browser
        CSS5 --> Browser
        CSS6 --> Browser
    end

    %% Storage and External
    subgraph "Storage & External Services"
        direction TB
        FaissIndex["Faiss Index (disk)"]:::storage
        MetadataDB["Metadata DB (PostgreSQL/SQLite)"]:::storage
        ExternalLLM["LLM API (OpenAI/HuggingFace)"]:::external
        Docker["Docker Compose"]:::external
        MetadataDB -->|queries| DjangoREST
        FaissIndex -->|load| FaissSearch
    end

    %% Connections
    CreateBin -->|"Faiss indices"| FaissIndex
    CreateBin -->|"Metadata DB"| MetadataDB
    Docker --> DjangoREST

    %% Click Events - Offline
    click Transnet "https://github.com/xndien2004/llm_powered_video_search/blob/main/data_extraction/TransnetV2/extract-frame-transnetv2.ipynb"
    click AudioExt "https://github.com/xndien2004/llm_powered_video_search/blob/main/data_extraction/audio/audio_extraction.py"
    click AudioDet "https://github.com/xndien2004/llm_powered_video_search/blob/main/data_extraction/audio/audio_detection.py"
    click AudioRec "https://github.com/xndien2004/llm_powered_video_search/blob/main/data_extraction/audio/audio_recognition.py"
    click MetadataNB "https://github.com/xndien2004/llm_powered_video_search/blob/main/data_extraction/metadata/aic24-ocr.ipynb"
    click MetadataNB "https://github.com/xndien2004/llm_powered_video_search/blob/main/data_extraction/metadata/color.ipynb"
    click MetadataNB "https://github.com/xndien2004/llm_powered_video_search/blob/main/data_extraction/metadata/aic-position-query.ipynb"
    click MetadataNB "https://github.com/xndien2004/llm_powered_video_search/blob/main/data_extraction/metadata/aic-caption.ipynb"
    click MetadataNB "https://github.com/xndien2004/llm_powered_video_search/blob/main/data_extraction/metadata/aic24-asr.ipynb"
    click MetadataNB "https://github.com/xndien2004/llm_powered_video_search/blob/main/data_extraction/metadata/aic24-asr-map.ipynb"
    click ImgFeat "https://github.com/xndien2004/llm_powered_video_search/blob/main/data_extraction/extract_image_features.ipynb"
    click TFIDF "https://github.com/xndien2004/llm_powered_video_search/blob/main/data_extraction/tfidf_encode.py"
    click CreateBin "https://github.com/xndien2004/llm_powered_video_search/blob/main/data_extraction/create_bin.py"

    %% Click Events - Backend
    click Config "https://github.com/xndien2004/llm_powered_video_search/blob/main/AIC/settings.py"
    click Config "https://github.com/xndien2004/llm_powered_video_search/blob/main/AIC/urls.py"
    click Config "https://github.com/xndien2004/llm_powered_video_search/blob/main/AIC/asgi.py"
    click Config "https://github.com/xndien2004/llm_powered_video_search/blob/main/AIC/wsgi.py"
    click DjangoREST "https://github.com/xndien2004/llm_powered_video_search/blob/main/app/viewAPI.py"
    click URLRouting "https://github.com/xndien2004/llm_powered_video_search/blob/main/app/urls.py"
    click URLRouting "https://github.com/xndien2004/llm_powered_video_search/blob/main/app/views.py"
    click ORMModels "https://github.com/xndien2004/llm_powered_video_search/blob/main/app/models.py"
    click Config "https://github.com/xndien2004/llm_powered_video_search/blob/main/manage.py"
    click Docker "https://github.com/xndien2004/llm_powered_video_search/blob/main/docker-compose.yml"
    click Docker "https://github.com/xndien2004/llm_powered_video_search/blob/main/requirements.txt"

    %% Click Events - Retrieval Engines
    click FaissSearch "https://github.com/xndien2004/llm_powered_video_search/blob/main/utils/faiss_search.py"
    click CombineSearch "https://github.com/xndien2004/llm_powered_video_search/blob/main/utils/combine_search.py"
    click LLMModule "https://github.com/xndien2004/llm_powered_video_search/blob/main/utils/LLM/llm.py"
    click LLMModule "https://github.com/xndien2004/llm_powered_video_search/blob/main/utils/LLM/llm_retrieval.py"
    click Langchain "https://github.com/xndien2004/llm_powered_video_search/blob/main/utils/langchain_search.py"
    click NLPProc "https://github.com/xndien2004/llm_powered_video_search/blob/main/utils/nlp_processing.py"
    click TagColor "https://github.com/xndien2004/llm_powered_video_search/blob/main/utils/smart_tagging.py"
    click VideoFuncs "https://github.com/xndien2004/llm_powered_video_search/blob/main/utils/video_retrieval/asr_retrieval.py"
    click VideoFuncs "https://github.com/xndien2004/llm_powered_video_search/blob/main/utils/video_retrieval/caption_retrieval.py"
    click VideoFuncs "https://github.com/xndien2004/llm_powered_video_search/blob/main/utils/video_retrieval/media_info_retrieval.py"
    click VideoFuncs "https://github.com/xndien2004/llm_powered_video_search/blob/main/utils/video_retrieval/object_retrieval.py"
    click VideoFuncs "https://github.com/xndien2004/llm_powered_video_search/blob/main/utils/video_retrieval/ocr_retrieval.py"
    click VideoFuncs "https://github.com/xndien2004/llm_powered_video_search/blob/main/utils/video_retrieval/tag_retrieval.py"

    %% Click Events - Frontend
    click HTMLIndex "https://github.com/xndien2004/llm_powered_video_search/blob/main/app/templates/index.html"
    click HTMLVideo "https://github.com/xndien2004/llm_powered_video_search/blob/main/app/templates/video.html"
    click JS1 "https://github.com/xndien2004/llm_powered_video_search/blob/main/app/static/js/jquery_colect.js"
    click JS2 "https://github.com/xndien2004/llm_powered_video_search/blob/main/app/static/js/others.js"
    click CSS1 "https://github.com/xndien2004/llm_powered_video_search/blob/main/app/static/css/index.css"
    click CSS2 "https://github.com/xndien2004/llm_powered_video_search/blob/main/app/static/css/main.css"
    click CSS3 "https://github.com/xndien2004/llm_powered_video_search/blob/main/app/static/css/chatbot.css"
    click CSS4 "https://github.com/xndien2004/llm_powered_video_search/blob/main/app/static/css/header.css"
    click CSS5 "https://github.com/xndien2004/llm_powered_video_search/blob/main/app/static/css/navLeft.css"
    click CSS6 "https://github.com/xndien2004/llm_powered_video_search/blob/main/app/static/css/navRight.css"

    %% Styles
    classDef offline fill:#ccffcc,stroke:#2f8f2f,color:#000
    classDef backend fill:#cce5ff,stroke:#2f4fa0,color:#000
    classDef retrieval fill:#e5ccff,stroke:#7a3fa0,color:#000
    classDef frontend fill:#ffe6cc,stroke:#a05a2f,color:#000
    classDef storage fill:#ffebcc,stroke:#a05800,color:#000
    classDef external fill:#fff2cc,stroke:#a08f00,color:#000
```
