How ssrajadh/sentrysearch works
1. What this repo is
SentrySearch is a Python 3.11+ CLI application for semantic search over video files. Its core architecture is an offline-first indexing pipeline plus a local vector database, with two interchangeable embedding backends: Google Gemini Embedding API or a local Qwen3-VL model.
2. Primary user-facing boundary: the CLI
sentrysearch/cli.py is the main entry point. It exposes commands for setup (init), indexing (index), querying (search), index maintenance (stats, remove, reset), and optional Tesla overlay rendering. The CLI orchestrates the rest of the system but does not hold the domain logic itself.
3. Ingestion pipeline: files → chunks → embeddings
The indexing flow starts by recursively discovering supported video files (.mp4, .mov). chunker.py splits each source video into overlapping time windows, with optional preprocessing to downscale and reduce frame rate before model ingestion. This is the first major performance boundary because it determines how much video is sent to the embedder and how many vector records are created.
4. Backend abstraction for video embeddings
The embedding layer is split into a small abstraction and concrete implementations:
- base_embedder.py defines the shared contract.
- gemini_embedder.py calls the Gemini API.
- local_embedder.py runs Qwen3-VL locally, with hardware-aware model selection and optional quantization.
- embedder.py appears to be the backend selection/facade layer.
This boundary matters because the rest of the system treats embeddings as backend-agnostic, while the index stores enough metadata to remember which backend/model created it.
5. Local persistence and index isolation
store.py manages the on-disk vector store, backed by ChromaDB per the README. It stores chunk embeddings plus metadata such as source path, chunk time range, and backend/model identity. Search auto-detects the correct backend/model from the stored index, which implies the store is also the authority for index compatibility and separation between models.
6. Search flow: text query → vector match → ranked results
search.py embeds the user’s text query into the same vector space as the indexed video chunks, then retrieves the nearest neighbors from the vector store. The output is a ranked list with similarity scores and time ranges. This is the second major system boundary: text input is transformed into retrieval over previously embedded video segments.
7. Clip generation: result → trimmed output file
trimmer.py takes the winning match, maps it back to the original source video, and extracts the corresponding time window into a standalone clip. The README makes clear this is usually automatic, but can be skipped with a flag. This stage depends on video tooling (ffmpeg or bundled imageio-ffmpeg) and is separate from search so retrieval and media extraction can evolve independently.
8. Optional Tesla-specific metadata overlay
Tesla dashcam footage can include embedded telemetry. dashcam.proto and dashcam_pb2.py define/consume that structured metadata, while metadata.py extracts it and overlay.py burns it into the trimmed clip as an on-video HUD. This is an optional post-processing branch after trimming, and it introduces an external dependency only for reverse geocoding when city/road names are desired.
9. Operational behavior and environment dependencies
The system is designed to run locally. It uses .env for API credentials, uv for installation/tooling, ChromaDB for persistence, and ffmpeg for decoding/trimming. The local backend additionally depends on GPU-capable ML runtimes (CUDA or Apple Metal) and, on macOS, a system FFmpeg installation for the video processor.
10. Main data flows to capture in an architecture diagram
There are two end-to-end flows worth diagramming:
- Indexing: video files on disk → chunking/preprocessing → embedding backend → local vector store.
- Searching: user text query → text embedding → vector retrieval from the same store → optional trim from original video → optional Tesla overlay → saved clip.
The key architectural boundary is that the search engine never transcribes video; it compares embeddings directly, and the persistence layer keeps backend/model-specific indexes isolated so incompatible embeddings are not mixed.
Main entry points: embedder.py Backend select · search.py Search · trimmer.py Trimmer · store.py Vector store
Components 19
CLI surface
User commands and orchestration entrypoint
- CLI · Main command dispatcher for init, index, search, and maintenance
sentrysearch/cli.py - Text query · Natural-language search prompt entered through the CLI
Indexing pipeline
Video discovery and chunk preparation before embedding
- Chunker · Finds supported videos and splits them into overlapping time windows
sentrysearch/chunker.py - Preprocess · Optional downscaling and frame-rate reduction before embedding
sentrysearch/chunker.py - Video files · Supported input videos discovered on disk
Embedding backends
Backend-agnostic embedding contract plus concrete implementations
- Embed API · Shared interface for turning video chunks or text into vectors
sentrysearch/base_embedder.py - Backend select · Chooses the active embedding backend and exposes a unified entrypoint
sentrysearch/embedder.py - Gemini · Calls the Google Gemini embedding API
sentrysearch/gemini_embedder.py - Local Qwen · Runs Qwen3-VL locally with hardware-aware model selection and optional quantization
sentrysearch/local_embedder.py
Local index
On-disk vector store and index compatibility boundaries
- Vector store · Persists chunk embeddings and metadata in a local ChromaDB-backed index
sentrysearch/store.py - Index metadata · Stores backend, model, source path, and chunk range needed to keep indexes isolated
sentrysearch/store.py
Retrieval and output
Search, trimming, and optional Tesla-specific overlay
- Search · Embeds a text query and retrieves nearest video chunks with scores
sentrysearch/search.py - Ranked hits · Similarity-ranked matches with source paths and time ranges
sentrysearch/search.py - Trimmer · Cuts the winning time window from the original source video
sentrysearch/trimmer.py - Tesla overlay · Optionally burns Tesla telemetry metadata into the trimmed clip
sentrysearch/overlay.py - Metadata · Extracts structured Tesla dashcam metadata for overlay rendering
sentrysearch/metadata.py - Dashcam proto · Protobuf schema and generated bindings for Tesla dashcam metadata
sentrysearch/dashcam.proto - Proto bindings · Generated protobuf module used by the metadata parser
sentrysearch/dashcam_pb2.py - Saved clip · Final trimmed clip, optionally with an overlay burned in
Connections 22
- CLI → Chunker: index · Starts indexing over discovered video files
- CLI → Search: search · Submits the user query to retrieval
- CLI → Trimmer: clip · Triggers extraction of a chosen result
- Video files → Chunker: scan · Recursively discovers supported video inputs
- Chunker → Preprocess: prepare · Optionally normalizes videos before embedding
- Preprocess → Backend select: chunks · Sends prepared chunks to the active backend
- Embed API → Backend select: contract · Defines the common embedding interface
- Backend select → Gemini: gemini · Routes embedding work to the cloud backend
- Backend select → Local Qwen: local · Routes embedding work to the local model backend
- Gemini → Vector store: store · Writes vectors and metadata to the index
- Local Qwen → Vector store: store · Writes vectors and metadata to the index
- Vector store → Index metadata: compat · Keeps backend and model identity with each index
- Text query → Search: embed query · Transforms text into the same vector space as indexed chunks
- Search → Vector store: retrieve · Fetches nearest neighbors from the local vector store
- Search → Ranked hits: rank · Produces scored matches and source time ranges
- Ranked hits → Trimmer: best hit · Uses the winning match to locate the source segment
- Trimmer → Saved clip: save · Extracts the segment into a standalone clip
- Metadata → Dashcam proto: parse · Interprets dashcam telemetry using the protobuf schema
- Dashcam proto → Proto bindings: bind · Generated protobuf bindings back the metadata reader
- Trimmer → Tesla overlay: optional · Hands trimmed video to the overlay stage when enabled
- Metadata → Tesla overlay: telemetry · Provides location and vehicle metadata for HUD rendering
- Tesla overlay → Saved clip: burn in · Produces the final annotated clip