How xndien2004/llm_powered_video_search works
Project type and purpose
• This is a full-stack web application: a multimedia/video retrieval system built with Django for the back end and plain HTML/CSS/JavaScript for the front end.
• Its purpose is to provide intelligent, multimodal video search (text, image, metadata) by combining offline data-extraction pipelines with online LLM-augmented retrieval.
File structure
• Top level:
• AIC/ ← Django project (settings, URLs, WSGI/ASGI)
• app/ ← Django “app” containing models, views, API endpoints, templates, static assets
• data_extraction/ ← offline scripts & notebooks for extracting frames, audio, metadata, TF-IDF encodings
• utils/ ← core retrieval logic: LLM retrieval, Faiss search, CLIP integration, audio/ocr/object tagging
• docker-compose.yml, manage.py, requirements.txt
• Architectural clues:
• app/viewAPI.py exposes REST endpoints that orchestrate calls into utils/
• data_extraction/* is an ETL pipeline run offline to populate indices and metadata stores
• Django ORM models in app/models.py back a relational database for metadata
README for additional insights
• Confirms technologies: Django, CLIP, Faiss, TF-IDF, PyTorch, OpenCV, Transformers
Main components
a. Data Extraction Pipeline (offline batch process)
• Raw Videos (source storage)
• TransnetV2 frame extraction
• Audio extraction & ASR
• TF-IDF encoding
→ Outputs: Feature files (NumPy/PyTorch tensors), metadata CSV/DB, Faiss indices
b. Backend (online service)
• Django REST API
• viewAPI.py orchestrator
• app/models.py → Relational DB (PostgreSQL/SQLite)
• Index store (Faiss on disk or in-memory)
c. Retrieval Engines (utils modules)
• CLIP-based image embedding + Faiss search
• TF-IDF text search
• Audio/ASR search
• OCR search
• LLM retriever (GPT-4 or local LLM via utils/LLM)
• combine_search merges scores
d. Frontend (client)
• HTML templates (index.html, video.html)
• JavaScript controllers (jquery_collect.js, others.js)
• CSS styling
e. External Services
• LLM API (OpenAI or Hugging Face)
• Docker for containerization (docker-compose)
Relationships and interactions
• Offline: Raw Videos → data_extraction pipeline → Feature store + Metadata DB + Faiss indices
• Online: Browser → Django URLs → viewAPI endpoint
→ calls retrieval engines in utils/
→ queries Metadata DB + Faiss indices + LLM API
→ aggregates & filters results
• Results returned as JSON → Frontend JS renders keyframes/video segments
Architectural patterns and design principles
• MVC pattern in Django (models/views/templates)
• Separation of offline ETL (data_extraction) vs. online retrieval (app + utils)
• Dependency injection: viewAPI imports modular utils search functions
• Indexing layer (Faiss) decoupled from relational metadata
• LLM integration as an external micro-service
Elements
• From raw video → extraction → indices
• From browser → Django → utils → DB/index → LLM → back to browser
• Green: Offline/batch processes
• Blue: Backend services
• Orange: Storage layers
• Yellow: External services
• Rectangles for services/modules