# tanishque-suthar/capstone architecture

> Architecture diagram of the GitHub repository tanishque-suthar/capstone, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tanishque-suthar/capstone
- Repository: https://github.com/tanishque-suthar/capstone

## Overview

1. **Top-level architecture**
   - This repo is a two-tier application: a **Python FastAPI backend** in `app/` and a **Next.js frontend** in `frontend/`.
   - The backend owns all video processing, persistence, and search logic; the frontend is a thin UI layer that calls backend APIs.

2. **Backend entrypoint and HTTP surface**
   - `app/main.py` is the backend bootstrap.
   - API routes are split by capability:
     - `app/routes/events.py` for event and video-related workflows
     - `app/routes/rag_routes.py` for semantic search and retrieval
     - `app/routes/privacy_routes.py` for privacy-related operations
   - This separation suggests the backend is organized as feature endpoints over a shared processing core.

3. **Core processing pipeline**
   - The pipeline package under `app/pipeline/` is the main engine:
     - `ingestion.py` brings video into the system
     - `perception.py` performs scene/event detection and object understanding
     - `privacy.py` applies privacy controls or redaction
     - `encryption.py` handles protected storage or secure transfer
     - `handoff.py` produces downstream-ready outputs or reports
     - `rag.py` handles retrieval/indexing for semantic search
   - The pipeline is likely orchestrated as a sequence of stages rather than a single monolith.

4. **Video intelligence flow**
   - The README indicates the main perception flow is:
     1. ingest video
     2. detect trigger events using **MOG2 background subtraction** and **entropy analysis**
     3. run **YOLOv11** for object detection
     4. track entities with **BoT-SORT**
   - This means the key data product of the backend is not raw video alone, but structured event/track metadata plus associated crops/clips.

5. **RAG / forensic search subsystem**
   - `app/pipeline/rag.py` plus `app/routes/rag_routes.py` implement semantic search over entity crops.
   - The README specifies **SigLIP embeddings** and **LanceDB** as the vector search stack.
   - The architectural boundary here is: visual crops are embedded offline/async, stored in a vector index, then queried through search endpoints for forensic retrieval.

6. **Privacy and security boundary**
   - `app/pipeline/privacy.py` and `app/routes/privacy_routes.py` indicate a dedicated privacy layer, likely for face blurring, masking, or redaction before sharing/export.
   - `app/pipeline/encryption.py` implies some artifacts may be encrypted at rest or before handoff.
   - The repo also includes a lightweight face model asset (`config/blaze_face_short_range.tflite`), reinforcing that privacy processing is a first-class subsystem, not an afterthought.

7. **Data persistence**
   - `app/database.py` and `app/models.py` point to a relational persistence layer for application state.
   - The README explicitly calls out **SQLite** as the event registry database.
   - The vector index is separate: **LanceDB** stored under `dataset/lancedb`, which means structured metadata and embedding search are persisted in different stores.

8. **Configuration and calibration assets**
   - The `config/` directory contains operational assets such as:
     - `homography.npy` / `homography_mat.py` for geometric calibration or coordinate mapping
     - `custom_tracker.yaml` for tracking configuration
     - `blaze_face_short_range.tflite` for on-device face detection
   - These files suggest the system depends on camera/scene calibration and model tuning, not just generic inference.

9. **Frontend application structure**
   - The frontend is a Next.js app with route-based pages for the main workflows:
     - upload (`/upload`)
     - event browsing and annotation (`/events/[id]`)
     - search (`/search`)
     - privacy (`/privacy`)
     - configuration (`/config`)
   - This is an operator console for reviewing processed video, not just a public-facing website.

10. **Frontend-to-backend coupling**
   - `frontend/src/lib/api.ts` is the client-side API bridge.
   - The frontend components such as `VideoAnnotator.tsx` and `Sidebar.tsx` suggest the UI is centered on:
     - inspecting event timelines
     - viewing tracks and crops
     - annotating or reviewing detections
   - The frontend does not appear to own business logic; it visualizes backend-derived artifacts.

11. **Primary runtime and tooling**
   - Backend: Python 3.10+, FastAPI, likely async API delivery with local model execution.
   - Frontend: Node.js 20+, Next.js 15+.
   - `ffmpeg` is an external runtime dependency for video processing/archival.
   - Model and ML dependencies are central to architecture: YOLO weights, SigLIP weights, and tracking models are loaded locally and may be downloaded on first run.

12. **Operational/data flow summary**
   - A typical flow is:
     - operator uploads or references a local video path
     - backend ingests and processes it
     - events, tracks, and crops are stored
     - privacy/encryption rules are applied as needed
     - embeddings are indexed into LanceDB
     - frontend displays the event and supports semantic search over the generated artifacts
   - So the main architectural boundary is between **raw video inputs** and **structured analytic outputs**.

13. **Auxiliary scripts and verification**
   - Files like `test_rag.py`, `debug_siglip*.py`, and `tmp_*` scripts indicate the repo also contains model/pipeline validation tooling.
   - These are not primary runtime components, but they show the team validates the perception and retrieval stack independently from the web app.

14. **What to draw in an architecture diagram**
   - The diagram should show:
     - **Frontend dashboard** calling the **FastAPI backend**
     - Backend split into **ingestion/perception/privacy/RAG/handoff**
     - Separate persistence for **SQLite metadata** and **LanceDB vector search**
     - External model/runtime dependencies: **YOLOv11**, **BoT-SORT**, **SigLIP**, **ffmpeg**
     - Local config/model assets under `config/` influencing inference and coordinate mapping
   - The most important conceptual split is between the **interactive UI**, the **video analytics pipeline**, and the **semantic retrieval store**.

Main entry points: [`app/pipeline/perception.py`](https://github.com/tanishque-suthar/capstone/blob/main/app/pipeline/perception.py) (Perception), [`app/main.py`](https://github.com/tanishque-suthar/capstone/blob/main/app/main.py) (App entry), [`frontend/src/app`](https://github.com/tanishque-suthar/capstone/tree/main/frontend/src/app) (UI shell), [`app/routes/events.py`](https://github.com/tanishque-suthar/capstone/blob/main/app/routes/events.py) (Events API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_backend["Backend"]
  node_backend_main["App entry<br/>FastAPI bootstrap<br/>[main.py]"]
  node_routes_events["Events API<br/>feature routes<br/>[events.py]"]
  node_routes_rag["Search API<br/>feature routes<br/>[rag_routes.py]"]
  node_routes_privacy["Privacy API<br/>feature routes<br/>[privacy_routes.py]"]
  node_ingestion["Ingestion<br/>pipeline stage<br/>[ingestion.py]"]
  node_perception["Perception<br/>pipeline stage<br/>[perception.py]"]
  node_privacy["Privacy<br/>pipeline stage<br/>[privacy.py]"]
  node_encryption["Encryption<br/>pipeline stage<br/>[encryption.py]"]
  node_handoff["Handoff<br/>pipeline stage<br/>[handoff.py]"]
  node_rag_pipeline["RAG core<br/>retrieval pipeline<br/>[rag.py]"]
end

subgraph group_frontend["Frontend"]
  node_frontend_app["UI shell<br/>nextjs app"]
  node_frontend_api["API client<br/>frontend bridge<br/>[api.ts]"]
  node_video_annotator["Annotator<br/>review ui<br/>[VideoAnnotator.tsx]"]
  node_sidebar["Sidebar<br/>navigation ui<br/>[Sidebar.tsx]"]
end

subgraph group_storage["Persistence"]
  node_database[("SQLite DB<br/>relational store<br/>[database.py]")]
  node_models["Data models<br/>schema layer<br/>[models.py]"]
  node_lancedb[("LanceDB<br/>vector store")]
end

subgraph group_assets["Assets"]
  node_config_assets["Config assets<br/>calibration/models"]
  node_yolo_bot_sort{{"Detectors<br/>ml stack"}}
  node_siglip_ffmpeg{{"Retrieval media<br/>ml/runtime"}}
end

node_frontend_app -->|"calls"| node_frontend_api
node_frontend_app -->|"renders"| node_video_annotator
node_frontend_app -->|"uses"| node_sidebar
node_frontend_api -->|"requests"| node_backend_main
node_backend_main -->|"mounts"| node_routes_events
node_backend_main -->|"mounts"| node_routes_rag
node_backend_main -->|"mounts"| node_routes_privacy
node_routes_events -->|"orchestrates"| node_ingestion
node_routes_events -->|"orchestrates"| node_perception
node_routes_privacy -->|"invokes"| node_privacy
node_routes_rag -->|"invokes"| node_rag_pipeline
node_ingestion -->|"feeds"| node_perception
node_perception -->|"produces"| node_privacy
node_perception -->|"emits"| node_handoff
node_privacy -->|"protects"| node_encryption
node_handoff -->|"stores metadata"| node_database
node_handoff -->|"indexes crops"| node_lancedb
node_rag_pipeline -->|"queries"| node_lancedb
node_backend_main -->|"uses"| node_models
node_perception -->|"depends on"| node_config_assets
node_perception -->|"uses"| node_yolo_bot_sort
node_rag_pipeline -->|"uses"| node_siglip_ffmpeg

click node_frontend_app "https://github.com/tanishque-suthar/capstone/tree/main/frontend/src/app"
click node_frontend_api "https://github.com/tanishque-suthar/capstone/blob/main/frontend/src/lib/api.ts"
click node_video_annotator "https://github.com/tanishque-suthar/capstone/blob/main/frontend/src/components/VideoAnnotator.tsx"
click node_sidebar "https://github.com/tanishque-suthar/capstone/blob/main/frontend/src/components/Sidebar.tsx"
click node_backend_main "https://github.com/tanishque-suthar/capstone/blob/main/app/main.py"
click node_routes_events "https://github.com/tanishque-suthar/capstone/blob/main/app/routes/events.py"
click node_routes_rag "https://github.com/tanishque-suthar/capstone/blob/main/app/routes/rag_routes.py"
click node_routes_privacy "https://github.com/tanishque-suthar/capstone/blob/main/app/routes/privacy_routes.py"
click node_ingestion "https://github.com/tanishque-suthar/capstone/blob/main/app/pipeline/ingestion.py"
click node_perception "https://github.com/tanishque-suthar/capstone/blob/main/app/pipeline/perception.py"
click node_privacy "https://github.com/tanishque-suthar/capstone/blob/main/app/pipeline/privacy.py"
click node_encryption "https://github.com/tanishque-suthar/capstone/blob/main/app/pipeline/encryption.py"
click node_handoff "https://github.com/tanishque-suthar/capstone/blob/main/app/pipeline/handoff.py"
click node_rag_pipeline "https://github.com/tanishque-suthar/capstone/blob/main/app/pipeline/rag.py"
click node_database "https://github.com/tanishque-suthar/capstone/blob/main/app/database.py"
click node_models "https://github.com/tanishque-suthar/capstone/blob/main/app/models.py"
click node_config_assets "https://github.com/tanishque-suthar/capstone/tree/main/config"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_backend_main,node_routes_events,node_routes_rag,node_routes_privacy,node_ingestion,node_perception,node_privacy,node_encryption,node_handoff,node_rag_pipeline toneBlue
class node_frontend_app,node_frontend_api,node_video_annotator,node_sidebar toneAmber
class node_database,node_models,node_lancedb toneMint
class node_config_assets,node_yolo_bot_sort,node_siglip_ffmpeg toneRose
```

## Components

### Backend

FastAPI service and video analytics pipeline

- [App entry](https://github.com/tanishque-suthar/capstone/blob/main/app/main.py): FastAPI bootstrap · Backend startup and application wiring. · `app/main.py`
- [Events API](https://github.com/tanishque-suthar/capstone/blob/main/app/routes/events.py): feature routes · Event and video workflow endpoints. · `app/routes/events.py`
- [Search API](https://github.com/tanishque-suthar/capstone/blob/main/app/routes/rag_routes.py): feature routes · Semantic retrieval and forensic search endpoints. · `app/routes/rag_routes.py`
- [Privacy API](https://github.com/tanishque-suthar/capstone/blob/main/app/routes/privacy_routes.py): feature routes · Privacy and redaction related endpoints. · `app/routes/privacy_routes.py`
- [Ingestion](https://github.com/tanishque-suthar/capstone/blob/main/app/pipeline/ingestion.py): pipeline stage · Brings video into the analytics pipeline. · `app/pipeline/ingestion.py`
- [Perception](https://github.com/tanishque-suthar/capstone/blob/main/app/pipeline/perception.py): pipeline stage · Scene, event, object, and track extraction from video. · `app/pipeline/perception.py`
- [Privacy](https://github.com/tanishque-suthar/capstone/blob/main/app/pipeline/privacy.py): pipeline stage · Applies masking, redaction, or privacy controls to artifacts. · `app/pipeline/privacy.py`
- [Encryption](https://github.com/tanishque-suthar/capstone/blob/main/app/pipeline/encryption.py): pipeline stage · Protects artifacts for storage or transfer. · `app/pipeline/encryption.py`
- [Handoff](https://github.com/tanishque-suthar/capstone/blob/main/app/pipeline/handoff.py): pipeline stage · Produces downstream-ready outputs and reports. · `app/pipeline/handoff.py`
- [RAG core](https://github.com/tanishque-suthar/capstone/blob/main/app/pipeline/rag.py): retrieval pipeline · Embeds and indexes visual crops for semantic search. · `app/pipeline/rag.py`

### Frontend

Next.js operator console

- [UI shell](https://github.com/tanishque-suthar/capstone/tree/main/frontend/src/app): nextjs app · Route-based operator interface for upload, review, search, privacy, and configuration. · `frontend/src/app`
- [API client](https://github.com/tanishque-suthar/capstone/blob/main/frontend/src/lib/api.ts): frontend bridge · Client-side API wrapper used by the UI to call backend endpoints. · `frontend/src/lib/api.ts`
- [Annotator](https://github.com/tanishque-suthar/capstone/blob/main/frontend/src/components/VideoAnnotator.tsx): review ui · Interactive event and track review experience for video artifacts. · `frontend/src/components/VideoAnnotator.tsx`
- [Sidebar](https://github.com/tanishque-suthar/capstone/blob/main/frontend/src/components/Sidebar.tsx): navigation ui · Primary navigation and workflow switching for the operator console. · `frontend/src/components/Sidebar.tsx`

### Persistence

Structured metadata and vector search stores

- [SQLite DB](https://github.com/tanishque-suthar/capstone/blob/main/app/database.py): relational store · Application metadata and event registry persistence. · `app/database.py`
- [Data models](https://github.com/tanishque-suthar/capstone/blob/main/app/models.py): schema layer · Persistent entity and record definitions used by the backend. · `app/models.py`
- **LanceDB**: vector store · Embedding index for visual retrieval and similarity search.

### Assets

Local model, calibration, and tracking configuration

- [Config assets](https://github.com/tanishque-suthar/capstone/tree/main/config): calibration/models · Homography, tracker settings, and local face model assets. · `config`
- **Detectors**: ml stack · YOLOv11 object detection and BoT-SORT tracking used in perception.
- **Retrieval media**: ml/runtime · SigLIP embeddings and ffmpeg video processing/runtime dependency.

## Connections

- UI shell → API client: calls · UI actions are routed through the client API layer.
- UI shell → Annotator: renders · The app mounts the annotation experience for event review.
- UI shell → Sidebar: uses · Navigation and workflow switching are provided by the sidebar.
- API client → App entry: requests · Frontend calls backend HTTP endpoints through a shared client wrapper.
- App entry → Events API: mounts · Event and video routes are registered on the FastAPI app.
- App entry → Search API: mounts · Search routes are registered on the FastAPI app.
- App entry → Privacy API: mounts · Privacy routes are registered on the FastAPI app.
- Events API → Ingestion: orchestrates · Event workflows trigger the ingest stage.
- Events API → Perception: orchestrates · Event workflows trigger perception and track extraction.
- Privacy API → Privacy: invokes · Privacy endpoints apply redaction and masking logic.
- Search API → RAG core: invokes · Search endpoints query the retrieval pipeline.
- Ingestion → Perception: feeds · Video ingestion hands off into perception.
- Perception → Privacy: produces · Detected artifacts can be passed through privacy controls.
- Perception → Handoff: emits · Structured events, tracks, and crops are prepared for downstream use.
- Privacy → Encryption: protects · Protected artifacts can be encrypted before storage or export.
- Handoff → SQLite DB: stores metadata · Event and processing state are persisted in SQLite.
- Handoff → LanceDB: indexes crops · Selected crops are embedded and stored for semantic search.
- RAG core → LanceDB: queries · Similarity search runs against the vector index.
- App entry → Data models: uses · Backend routes and pipeline stages share common data models.
- Perception → Config assets: depends on · Calibration and tracker/model assets shape inference and tracking.
- Perception → Detectors: uses · Object detection and multi-object tracking depend on the local ML stack.
- RAG core → Retrieval media: uses · Retrieval relies on embedding models and media processing runtime.
