# rishabhjain2003/indian-traffic-sign-dataset architecture

> Architecture diagram of the GitHub repository rishabhjain2003/indian-traffic-sign-dataset, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rishabhjain2003/indian-traffic-sign-dataset
- Repository: https://github.com/rishabhjain2003/indian-traffic-sign-dataset

## Overview

1. **What this repository is**
   - This is an offline data-ingestion and inference pipeline for building an Indian traffic-sign dataset from YouTube dashcam footage.
   - The architecture is centered on a local Python process that discovers videos, processes them in parallel, and persists results in PostgreSQL.

2. **Top-level control plane**
   - `pipeline.py` is the main orchestrator and CLI entrypoint.
   - It coordinates three modes: search-and-process, workers-only resume, and single-video processing.
   - It also owns thread lifecycle, logging, shutdown behavior, and the producer/consumer handoff.

3. **Discovery subsystem**
   - `src/scraper.py` is the producer.
   - It uses YouTube search via `pytubefix` InnerTube API, driven by keyword lists from `combinatorial_keywords.txt` or `urls.txt`.
   - Its job is to discover candidate videos and immediately persist them into the database as work items, rather than handing them directly to workers in memory.

4. **Persistent work queue boundary**
   - `discovered_videos` in PostgreSQL is the system’s durable queue.
   - It tracks lifecycle state: `pending`, `processing`, `completed`, `failed`.
   - This queue is the main boundary between discovery and processing, which makes the pipeline resumable across crashes and restarts.

5. **Parallel processing subsystem**
   - Worker threads in `pipeline.py` consume from the queue using atomic claim semantics.
   - The README indicates `FOR UPDATE SKIP LOCKED` is used so multiple consumers can safely process different rows without contention.
   - This is the core scalability mechanism: discovery runs in one thread, while N workers process videos concurrently.

6. **Video ingestion path**
   - `src/downloader.py` fetches the selected YouTube video.
   - It handles progressive or adaptive streams and uses `ffmpeg` when muxing is needed for higher-quality outputs.
   - The output of this stage is a temporary local video file under `output/videos/`.

7. **Frame generation path**
   - `src/extractor.py` converts each downloaded video into frame images using `ffmpeg` as a subprocess.
   - Frame rate and stride are configurable, so this stage controls storage volume and inference cost.
   - Frames are written to `output/frames/<video_id>/` and are treated as temporary artifacts.

8. **Detection subsystem**
   - `src/detector.py` runs YOLOv8 inference using Ultralytics against the extracted frames.
   - It supports batch inference, configurable confidence/IoU thresholds, and optional annotated output frames.
   - The detector is the bridge from raw imagery to structured traffic-sign observations.

9. **Persistence and data model**
   - `src/db.py` handles PostgreSQL schema creation and CRUD operations.
   - The normalized result model is:
     - `videos` for video metadata,
     - `detected_frames` for frames that contain at least one sign,
     - `detections` for individual bounding boxes,
     - `discovered_videos` as the durable queue.
   - This separation matters: raw metadata, frame-level hits, and box-level detections are stored independently for later querying.

10. **Cleanup and storage policy**
   - Processed videos and raw frames are deleted after use by default.
   - Only annotated detection frames are retained in `output/detections/<video_id>/`.
   - This makes the pipeline storage-efficient for large-scale batch runs, while preserving human-reviewable outputs.

11. **Supporting utilities**
   - `keyword_generator.py` generates the combinatorial search terms used by the scraper.
   - `scripts/query_appearances.py` is a downstream analytics tool that queries the database for continuous appearance ranges per sign class and exports `output/appearances.csv`.
   - These are not part of the core runtime loop, but they support dataset expansion and result analysis.

12. **Configuration and operational boundaries**
   - `config/settings.py` centralizes runtime settings, overridden by environment variables.
   - The main external dependencies that materially shape the architecture are: PostgreSQL for durable state, FFmpeg for media handling, and YOLOv8 weights (`best.pt`) for detection.
   - The pipeline is designed to be long-running, restartable, and machine-local rather than a cloud-native distributed service.

13. **End-to-end data flow**
   - Search keywords or URLs → YouTube discovery → `discovered_videos` queue → worker claim → download → frame extraction → YOLO detection → structured DB inserts → optional annotated images → cleanup.
   - If you are drawing the architecture, the most important diagram boundaries are: discovery vs processing, ephemeral filesystem vs persistent PostgreSQL, and orchestration vs worker execution.

Main entry points: [`pipeline.py`](https://github.com/rishabhjain2003/indian-traffic-sign-dataset/blob/main/pipeline.py) (Pipeline), [`src/db.py`](https://github.com/rishabhjain2003/indian-traffic-sign-dataset/blob/main/src/db.py) (Database), [`src/downloader.py`](https://github.com/rishabhjain2003/indian-traffic-sign-dataset/blob/main/src/downloader.py) (Downloader), [`src/detector.py`](https://github.com/rishabhjain2003/indian-traffic-sign-dataset/blob/main/src/detector.py) (Detector)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_orchestration["Orchestration"]
  node_pipeline["Pipeline<br/>orchestrator/cli<br/>[pipeline.py]"]
  node_settings["Settings<br/>config<br/>[settings.py]"]
end

subgraph group_group_discovery["Discovery"]
  node_scraper["Scraper<br/>discovery producer<br/>[scraper.py]"]
  node_keywords["Keywords<br/>search inputs"]
  node_youtube(("YouTube<br/>external source"))
end

subgraph group_group_processing["Processing"]
  node_downloader["Downloader<br/>media fetch<br/>[downloader.py]"]
  node_extractor["Extractor<br/>frame extraction<br/>[extractor.py]"]
  node_detector["Detector<br/>yolov8 inference<br/>[detector.py]"]
  node_ffmpeg(("FFmpeg<br/>media tool"))
  node_yolo(("YOLOv8<br/>model runtime"))
  node_weights["Weights<br/>model artifact"]
end

subgraph group_group_storage["Storage"]
  node_queue["Queue"]
  node_db[("Database<br/>postgres access<br/>[db.py]")]
  node_videos[("Videos<br/>video metadata")]
  node_frames[("Frame hits<br/>frame records")]
  node_detections[("Detections<br/>bbox records")]
end

subgraph group_group_support["Support"]
  node_keywordgen["Keyword Gen<br/>support tool"]
  node_appearances["Appearances<br/>analytics script"]
end

node_pipeline -->|"start discovery"| node_scraper
node_pipeline -->|"manage state"| node_db
node_pipeline -->|"worker flow"| node_downloader
node_pipeline -->|"process video"| node_extractor
node_pipeline -->|"run inference"| node_detector
node_settings -->|"configure"| node_pipeline
node_settings -->|"configure"| node_scraper
node_keywords -->|"seed search"| node_scraper
node_keywordgen -->|"generate"| node_keywords
node_scraper -->|"search"| node_youtube
node_scraper -->|"enqueue"| node_queue
node_queue -->|"claim"| node_pipeline
node_queue -->|"dequeue"| node_downloader
node_downloader -->|"fetch"| node_youtube
node_downloader -->|"mux"| node_ffmpeg
node_downloader -->|"hand off"| node_extractor
node_extractor -->|"slice frames"| node_ffmpeg
node_extractor -->|"frames"| node_detector
node_detector -->|"infer"| node_yolo
node_detector -->|"load"| node_weights
node_detector -->|"persist results"| node_db
node_db -->|"store metadata"| node_videos
node_db -->|"store hits"| node_frames
node_db -->|"store boxes"| node_detections
node_db -->|"state updates"| node_queue
node_appearances -->|"query"| node_db

click node_pipeline "https://github.com/rishabhjain2003/indian-traffic-sign-dataset/blob/main/pipeline.py"
click node_settings "https://github.com/rishabhjain2003/indian-traffic-sign-dataset/blob/main/config/settings.py"
click node_scraper "https://github.com/rishabhjain2003/indian-traffic-sign-dataset/blob/main/src/scraper.py"
click node_db "https://github.com/rishabhjain2003/indian-traffic-sign-dataset/blob/main/src/db.py"
click node_downloader "https://github.com/rishabhjain2003/indian-traffic-sign-dataset/blob/main/src/downloader.py"
click node_extractor "https://github.com/rishabhjain2003/indian-traffic-sign-dataset/blob/main/src/extractor.py"
click node_detector "https://github.com/rishabhjain2003/indian-traffic-sign-dataset/blob/main/src/detector.py"
click node_keywordgen "https://github.com/rishabhjain2003/indian-traffic-sign-dataset/blob/main/keyword_generator.py"
click node_appearances "https://github.com/rishabhjain2003/indian-traffic-sign-dataset/blob/main/scripts/query_appearances.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pipeline,node_settings toneBlue
class node_scraper,node_keywords,node_youtube toneAmber
class node_downloader,node_extractor,node_detector,node_ffmpeg,node_yolo,node_weights toneMint
class node_queue,node_db,node_videos,node_frames,node_detections toneRose
class node_keywordgen,node_appearances toneIndigo
```

## Components

### Orchestration

CLI entrypoint, runtime control, and shared configuration.

- [Pipeline](https://github.com/rishabhjain2003/indian-traffic-sign-dataset/blob/main/pipeline.py): orchestrator/cli · Main entrypoint that coordinates discovery, worker threads, resume modes, and shutdown. · `pipeline.py`
- [Settings](https://github.com/rishabhjain2003/indian-traffic-sign-dataset/blob/main/config/settings.py): config · Central runtime configuration overridden by environment variables. · `config/settings.py`

### Discovery

Keyword-driven video discovery and queueing.

- [Scraper](https://github.com/rishabhjain2003/indian-traffic-sign-dataset/blob/main/src/scraper.py): discovery producer · Searches YouTube and persists discovered videos as durable work items. · `src/scraper.py`
- **Keywords**: search inputs · Keyword and URL inputs used to seed discovery.
- **YouTube**: external source · Remote video source discovered through search and fetched by the downloader.

### Processing

Download, frame extraction, and detection stages.

- [Downloader](https://github.com/rishabhjain2003/indian-traffic-sign-dataset/blob/main/src/downloader.py): media fetch · Downloads YouTube videos and muxes streams when needed. · `src/downloader.py`
- [Extractor](https://github.com/rishabhjain2003/indian-traffic-sign-dataset/blob/main/src/extractor.py): frame extraction · Uses ffmpeg to turn videos into frame images at configurable rate and stride. · `src/extractor.py`
- [Detector](https://github.com/rishabhjain2003/indian-traffic-sign-dataset/blob/main/src/detector.py): yolov8 inference · Runs Ultralytics YOLOv8 inference on extracted frames and can emit annotated outputs. · `src/detector.py`
- **FFmpeg**: media tool · Subprocess used for muxing and frame extraction.
- **YOLOv8**: model runtime · Ultralytics inference backend using the best.pt weights.
- **Weights**: model artifact · Trained detection weights consumed by the detector.

### Storage

PostgreSQL schema and durable state.

- **Queue**: durable work queue · PostgreSQL-backed discovered_videos queue with pending, processing, completed, and failed states.
- [Database](https://github.com/rishabhjain2003/indian-traffic-sign-dataset/blob/main/src/db.py): postgres access · Schema creation and CRUD boundary for persistent results and work state. · `src/db.py`
- **Videos**: video metadata · Normalized table for video-level metadata and processing status.
- **Frame hits**: frame records · Frames that contain at least one detected traffic sign.
- **Detections**: bbox records · Individual bounding-box detections for traffic signs.

### Support

Keyword generation and downstream analysis tools.

- [Keyword Gen](https://github.com/rishabhjain2003/indian-traffic-sign-dataset/blob/main/keyword_generator.py): support tool · Generates combinatorial search terms for discovery. · `keyword_generator.py`
- [Appearances](https://github.com/rishabhjain2003/indian-traffic-sign-dataset/blob/main/scripts/query_appearances.py): analytics script · Queries detection ranges and exports appearance analytics. · `scripts/query_appearances.py`

## Connections

- Pipeline → Scraper: start discovery · Launches producer mode to discover videos.
- Pipeline → Database: manage state · Reads and updates persistent work state and results.
- Pipeline → Downloader: worker flow · Coordinates worker execution for claimed queue items.
- Pipeline → Extractor: process video · Runs the media pipeline for a claimed video.
- Pipeline → Detector: run inference · Invokes detection over extracted frames.
- Settings → Pipeline: configure · Provides runtime settings for orchestration and worker behavior.
- Settings → Scraper: configure · Supplies search and discovery settings.
- Keywords → Scraper: seed search · Feeds search terms and URLs into discovery.
- Keyword Gen → Keywords: generate · Produces combinatorial search terms for scraper input.
- Scraper → YouTube: search · Discovers candidate videos via YouTube InnerTube search.
- Scraper → Queue: enqueue · Persists discovered videos as durable work items.
- Queue → Pipeline: claim · Workers atomically claim pending rows using skip-locked semantics.
- Queue → Downloader: dequeue · Provides the next video to process.
- Downloader → YouTube: fetch · Downloads the selected source video.
- Downloader → FFmpeg: mux · Uses FFmpeg when stream muxing is required.
- Downloader → Extractor: hand off · Passes downloaded video files to frame extraction.
- Extractor → FFmpeg: slice frames · Extracts frames with FFmpeg subprocesses.
- Extractor → Detector: frames · Sends extracted frames to the inference stage.
- Detector → YOLOv8: infer · Runs batch YOLOv8 inference on frames.
- Detector → Weights: load · Consumes the trained detection weights.
- Detector → Database: persist results · Writes frame hits and bounding-box detections.
- Database → Videos: store metadata · Maintains video-level records.
- Database → Frame hits: store hits · Stores frames with at least one detection.
- Database → Detections: store boxes · Stores individual detection boxes.
- Database → Queue: state updates · Tracks queue lifecycle transitions.
- Appearances → Database: query · Reads detections for appearance-range analytics.