How tanishque-suthar/capstone works
1. Top-level architecture
- This repo is a two-tier application: a Python FastAPI backend in app/ and a Next.js frontend in frontend/.
- The backend owns all video processing, persistence, and search logic; the frontend is a thin UI layer that calls backend APIs.
2. Backend entrypoint and HTTP surface
- app/main.py is the backend bootstrap.
- API routes are split by capability:
- app/routes/events.py for event and video-related workflows
- app/routes/rag_routes.py for semantic search and retrieval
- app/routes/privacy_routes.py for privacy-related operations
- This separation suggests the backend is organized as feature endpoints over a shared processing core.
3. Core processing pipeline
- The pipeline package under app/pipeline/ is the main engine:
- ingestion.py brings video into the system
- perception.py performs scene/event detection and object understanding
- privacy.py applies privacy controls or redaction
- encryption.py handles protected storage or secure transfer
- handoff.py produces downstream-ready outputs or reports
- rag.py handles retrieval/indexing for semantic search
- The pipeline is likely orchestrated as a sequence of stages rather than a single monolith.
4. Video intelligence flow
- The README indicates the main perception flow is:
1. ingest video
2. detect trigger events using MOG2 background subtraction and entropy analysis
3. run YOLOv11 for object detection
4. track entities with BoT-SORT
- This means the key data product of the backend is not raw video alone, but structured event/track metadata plus associated crops/clips.
5. RAG / forensic search subsystem
- app/pipeline/rag.py plus app/routes/rag_routes.py implement semantic search over entity crops.
- The README specifies SigLIP embeddings and LanceDB as the vector search stack.
- The architectural boundary here is: visual crops are embedded offline/async, stored in a vector index, then queried through search endpoints for forensic retrieval.
6. Privacy and security boundary
- app/pipeline/privacy.py and app/routes/privacy_routes.py indicate a dedicated privacy layer, likely for face blurring, masking, or redaction before sharing/export.
- app/pipeline/encryption.py implies some artifacts may be encrypted at rest or before handoff.
- The repo also includes a lightweight face model asset (config/blaze_face_short_range.tflite), reinforcing that privacy processing is a first-class subsystem, not an afterthought.
7. Data persistence
- app/database.py and app/models.py point to a relational persistence layer for application state.
- The README explicitly calls out SQLite as the event registry database.
- The vector index is separate: LanceDB stored under dataset/lancedb, which means structured metadata and embedding search are persisted in different stores.
8. Configuration and calibration assets
- The config/ directory contains operational assets such as:
- homography.npy / homography_mat.py for geometric calibration or coordinate mapping
- custom_tracker.yaml for tracking configuration
- blaze_face_short_range.tflite for on-device face detection
- These files suggest the system depends on camera/scene calibration and model tuning, not just generic inference.
9. Frontend application structure
- The frontend is a Next.js app with route-based pages for the main workflows:
- upload (/upload)
- event browsing and annotation (/events/[id])
- search (/search)
- privacy (/privacy)
- configuration (/config)
- This is an operator console for reviewing processed video, not just a public-facing website.
10. Frontend-to-backend coupling
- frontend/src/lib/api.ts is the client-side API bridge.
- The frontend components such as VideoAnnotator.tsx and Sidebar.tsx suggest the UI is centered on:
- inspecting event timelines
- viewing tracks and crops
- annotating or reviewing detections
- The frontend does not appear to own business logic; it visualizes backend-derived artifacts.
11. Primary runtime and tooling
- Backend: Python 3.10+, FastAPI, likely async API delivery with local model execution.
- Frontend: Node.js 20+, Next.js 15+.
- ffmpeg is an external runtime dependency for video processing/archival.
- Model and ML dependencies are central to architecture: YOLO weights, SigLIP weights, and tracking models are loaded locally and may be downloaded on first run.
12. Operational/data flow summary
- A typical flow is:
- operator uploads or references a local video path
- backend ingests and processes it
- events, tracks, and crops are stored
- privacy/encryption rules are applied as needed
- embeddings are indexed into LanceDB
- frontend displays the event and supports semantic search over the generated artifacts
- So the main architectural boundary is between raw video inputs and structured analytic outputs.
13. Auxiliary scripts and verification
- Files like test_rag.py, debug_siglip*.py, and tmp_* scripts indicate the repo also contains model/pipeline validation tooling.
- These are not primary runtime components, but they show the team validates the perception and retrieval stack independently from the web app.
14. What to draw in an architecture diagram
- The diagram should show:
- Frontend dashboard calling the FastAPI backend
- Backend split into ingestion/perception/privacy/RAG/handoff
- Separate persistence for SQLite metadata and LanceDB vector search
- External model/runtime dependencies: YOLOv11, BoT-SORT, SigLIP, ffmpeg
- Local config/model assets under config/ influencing inference and coordinate mapping
- The most important conceptual split is between the interactive UI, the video analytics pipeline, and the semantic retrieval store.
Main entry points: perception.py Perception · main.py App entry · app UI shell · events.py Events API
Components 20
Backend
FastAPI service and video analytics pipeline
- App entry · Backend startup and application wiring.
app/main.py - Events API · Event and video workflow endpoints.
app/routes/events.py - Search API · Semantic retrieval and forensic search endpoints.
app/routes/rag_routes.py - Privacy API · Privacy and redaction related endpoints.
app/routes/privacy_routes.py - Ingestion · Brings video into the analytics pipeline.
app/pipeline/ingestion.py - Perception · Scene, event, object, and track extraction from video.
app/pipeline/perception.py - Privacy · Applies masking, redaction, or privacy controls to artifacts.
app/pipeline/privacy.py - Encryption · Protects artifacts for storage or transfer.
app/pipeline/encryption.py - Handoff · Produces downstream-ready outputs and reports.
app/pipeline/handoff.py - RAG core · Embeds and indexes visual crops for semantic search.
app/pipeline/rag.py
Frontend
Next.js operator console
- UI shell · Route-based operator interface for upload, review, search, privacy, and configuration.
frontend/src/app - API client · Client-side API wrapper used by the UI to call backend endpoints.
frontend/src/lib/api.ts - Annotator · Interactive event and track review experience for video artifacts.
frontend/src/components/VideoAnnotator.tsx - Sidebar · Primary navigation and workflow switching for the operator console.
frontend/src/components/Sidebar.tsx
Persistence
Structured metadata and vector search stores
- SQLite DB · Application metadata and event registry persistence.
app/database.py - Data models · Persistent entity and record definitions used by the backend.
app/models.py - LanceDB · Embedding index for visual retrieval and similarity search.
Assets
Local model, calibration, and tracking configuration
- Config assets · Homography, tracker settings, and local face model assets.
config - Detectors · YOLOv11 object detection and BoT-SORT tracking used in perception.
- Retrieval media · SigLIP embeddings and ffmpeg video processing/runtime dependency.
Connections 22
- UI shell → API client: calls · UI actions are routed through the client API layer.
- UI shell → Annotator: renders · The app mounts the annotation experience for event review.
- UI shell → Sidebar: uses · Navigation and workflow switching are provided by the sidebar.
- API client → App entry: requests · Frontend calls backend HTTP endpoints through a shared client wrapper.
- App entry → Events API: mounts · Event and video routes are registered on the FastAPI app.
- App entry → Search API: mounts · Search routes are registered on the FastAPI app.
- App entry → Privacy API: mounts · Privacy routes are registered on the FastAPI app.
- Events API → Ingestion: orchestrates · Event workflows trigger the ingest stage.
- Events API → Perception: orchestrates · Event workflows trigger perception and track extraction.
- Privacy API → Privacy: invokes · Privacy endpoints apply redaction and masking logic.
- Search API → RAG core: invokes · Search endpoints query the retrieval pipeline.
- Ingestion → Perception: feeds · Video ingestion hands off into perception.
- Perception → Privacy: produces · Detected artifacts can be passed through privacy controls.
- Perception → Handoff: emits · Structured events, tracks, and crops are prepared for downstream use.
- Privacy → Encryption: protects · Protected artifacts can be encrypted before storage or export.
- Handoff → SQLite DB: stores metadata · Event and processing state are persisted in SQLite.
- Handoff → LanceDB: indexes crops · Selected crops are embedded and stored for semantic search.
- RAG core → LanceDB: queries · Similarity search runs against the vector index.
- App entry → Data models: uses · Backend routes and pipeline stages share common data models.
- Perception → Config assets: depends on · Calibration and tracker/model assets shape inference and tracking.
- Perception → Detectors: uses · Object detection and multi-object tracking depend on the local ML stack.
- RAG core → Retrieval media: uses · Retrieval relies on embedding models and media processing runtime.