How robertmpevec/clip_genius works
Project type and purpose
• This is a full-stack, video-processing and analytics application called “Scoreboard Analyzer AI.”
• Purpose: ingest raw sports video, detect and track scoreboard changes via OCR, analyze commentary with NLP to rank exciting moments, generate short highlight clips, and serve them through a web UI.
File structure
• Top-level folders:
• backend/ – Python modules for processing, services, models, database and main API entrypoint (main.py)
• frontend/ – web UI code: app.js, static assets and a Python “clip_genius_application.py” (likely orchestration or notebook)
• deployment/ – Dockerfile and Kubernetes manifest
• database files under backend/database (SQL schema, db_handler.py)
• tests/ – unit and integration tests
• Architectural cues:
• backend/Processing – image pre-processing, OCR wrapper, parallelization
• backend/services – separate scripts for clip merging, vector DB (FAISS), search/classification
• backend/models – model utility functions (likely wrapping NeMo NLP models)
Pull insights from README
• Core tech: OpenCV, PyTesseract OCR, NVIDIA NeMo NLP, FFmpeg, FAISS vector DB
• Workflow: video → OCR scoreboard detection → NLP commentary analysis → clip ranking → clip generation
• Targets real-time score tracking and automated highlight creation
Main components
a. Video Ingestion
• Component: “Video Input Service”
• In: raw video files (clips/ directory or streaming endpoint)
b. Pre-Processing & OCR
• backend/Processing/PreProcessing.py & ImageProcessingFunctions.py
• Calls: OpenCV → frame extraction, normalization → PyTesseract OCR → scoreboard text
c. NLP Commentary Analysis
• backend/models/model_utils.py + NeMo models
• Input: audio transcript or text feed → output: excitement score
d. Clip Generation Service
• backend/services/process_video.py + FFmpeg calls
• Merges top-ranked segments (see merged_video.mp4)
e. Data Stores
• Relational DB (backend/database/database-schema.sql + db_handler.py) → stores timestamped scores
• FAISS Vector DB (backend/services/vector_database.py + highlight_vectors.faiss) → nearest-neighbor search on NLP embeddings
f. API Layer / Orchestration
• backend/main.py → REST endpoints for: submit video, get scoreboard timeline, fetch highlights
g. Frontend UI
• frontend/app.js + static/PresentationFormatter.js, ReportsFormatter.js
• Interacts via AJAX/REST with backend/main.py
h. Deployment & Infrastructure
• Docker container per service (monolithic backend + frontend or separate containers)
• Kubernetes manifest (deployment/k8s.yaml) for scaling, service discovery
Relationships and interactions (data flow)
1. User uploads or points to video → sent to Video Input Service
2. Video frames → Pre-Processing → OCR → Scoreboard DB writes
3. Transcript feed → NLP Service → embeddings stored in FAISS
4. Backend scheduler or user request → Clip Generation Service pulls high-excitement segments → produces merged highlights
5. Frontend UI fetches scoreboard data and highlight URLs via REST → renders video player and stats
Architectural patterns & design principles
• Pipeline pattern: clearly separated sequential stages (Pre-Processing → OCR → NLP → Clip Gen)
• Modular/service-oriented backend: distinct Python modules under services/ for clipping, vector search, classification
• Data persistence separation: relational vs. vector DB for low-latency similarity search
• Containerization + orchestration for reproducibility and scalability