How rishabhjain2003/indian-traffic-sign-dataset works
1. What this repository is
- This is an offline data-ingestion and inference pipeline for building an Indian traffic-sign dataset from YouTube dashcam footage.
- The architecture is centered on a local Python process that discovers videos, processes them in parallel, and persists results in PostgreSQL.
2. Top-level control plane
- pipeline.py is the main orchestrator and CLI entrypoint.
- It coordinates three modes: search-and-process, workers-only resume, and single-video processing.
- It also owns thread lifecycle, logging, shutdown behavior, and the producer/consumer handoff.
3. Discovery subsystem
- src/scraper.py is the producer.
- It uses YouTube search via pytubefix InnerTube API, driven by keyword lists from combinatorial_keywords.txt or urls.txt.
- Its job is to discover candidate videos and immediately persist them into the database as work items, rather than handing them directly to workers in memory.
4. Persistent work queue boundary
- discovered_videos in PostgreSQL is the system’s durable queue.
- It tracks lifecycle state: pending, processing, completed, failed.
- This queue is the main boundary between discovery and processing, which makes the pipeline resumable across crashes and restarts.
5. Parallel processing subsystem
- Worker threads in pipeline.py consume from the queue using atomic claim semantics.
- The README indicates FOR UPDATE SKIP LOCKED is used so multiple consumers can safely process different rows without contention.
- This is the core scalability mechanism: discovery runs in one thread, while N workers process videos concurrently.
6. Video ingestion path
- src/downloader.py fetches the selected YouTube video.
- It handles progressive or adaptive streams and uses ffmpeg when muxing is needed for higher-quality outputs.
- The output of this stage is a temporary local video file under output/videos/.
7. Frame generation path
- src/extractor.py converts each downloaded video into frame images using ffmpeg as a subprocess.
- Frame rate and stride are configurable, so this stage controls storage volume and inference cost.
- Frames are written to output/frames/<video_id>/ and are treated as temporary artifacts.
8. Detection subsystem
- src/detector.py runs YOLOv8 inference using Ultralytics against the extracted frames.
- It supports batch inference, configurable confidence/IoU thresholds, and optional annotated output frames.
- The detector is the bridge from raw imagery to structured traffic-sign observations.
9. Persistence and data model
- src/db.py handles PostgreSQL schema creation and CRUD operations.
- The normalized result model is:
- videos for video metadata,
- detected_frames for frames that contain at least one sign,
- detections for individual bounding boxes,
- discovered_videos as the durable queue.
- This separation matters: raw metadata, frame-level hits, and box-level detections are stored independently for later querying.
10. Cleanup and storage policy
- Processed videos and raw frames are deleted after use by default.
- Only annotated detection frames are retained in output/detections/<video_id>/.
- This makes the pipeline storage-efficient for large-scale batch runs, while preserving human-reviewable outputs.
11. Supporting utilities
- keyword_generator.py generates the combinatorial search terms used by the scraper.
- scripts/query_appearances.py is a downstream analytics tool that queries the database for continuous appearance ranges per sign class and exports output/appearances.csv.
- These are not part of the core runtime loop, but they support dataset expansion and result analysis.
12. Configuration and operational boundaries
- config/settings.py centralizes runtime settings, overridden by environment variables.
- The main external dependencies that materially shape the architecture are: PostgreSQL for durable state, FFmpeg for media handling, and YOLOv8 weights (best.pt) for detection.
- The pipeline is designed to be long-running, restartable, and machine-local rather than a cloud-native distributed service.
13. End-to-end data flow
- Search keywords or URLs → YouTube discovery → discovered_videos queue → worker claim → download → frame extraction → YOLO detection → structured DB inserts → optional annotated images → cleanup.
- If you are drawing the architecture, the most important diagram boundaries are: discovery vs processing, ephemeral filesystem vs persistent PostgreSQL, and orchestration vs worker execution.
Main entry points: pipeline.py Pipeline · db.py Database · downloader.py Downloader · detector.py Detector
Components 18
Orchestration
CLI entrypoint, runtime control, and shared configuration.
Discovery
Keyword-driven video discovery and queueing.
- Scraper · Searches YouTube and persists discovered videos as durable work items.
src/scraper.py - Keywords · Keyword and URL inputs used to seed discovery.
- YouTube · Remote video source discovered through search and fetched by the downloader.
Processing
Download, frame extraction, and detection stages.
- Downloader · Downloads YouTube videos and muxes streams when needed.
src/downloader.py - Extractor · Uses ffmpeg to turn videos into frame images at configurable rate and stride.
src/extractor.py - Detector · Runs Ultralytics YOLOv8 inference on extracted frames and can emit annotated outputs.
src/detector.py - FFmpeg · Subprocess used for muxing and frame extraction.
- YOLOv8 · Ultralytics inference backend using the best.pt weights.
- Weights · Trained detection weights consumed by the detector.
Storage
PostgreSQL schema and durable state.
- Queue · PostgreSQL-backed discovered_videos queue with pending, processing, completed, and failed states.
- Database · Schema creation and CRUD boundary for persistent results and work state.
src/db.py - Videos · Normalized table for video-level metadata and processing status.
- Frame hits · Frames that contain at least one detected traffic sign.
- Detections · Individual bounding-box detections for traffic signs.
Support
Keyword generation and downstream analysis tools.
- Keyword Gen · Generates combinatorial search terms for discovery.
keyword_generator.py - Appearances · Queries detection ranges and exports appearance analytics.
scripts/query_appearances.py
Connections 26
- Pipeline → Scraper: start discovery · Launches producer mode to discover videos.
- Pipeline → Database: manage state · Reads and updates persistent work state and results.
- Pipeline → Downloader: worker flow · Coordinates worker execution for claimed queue items.
- Pipeline → Extractor: process video · Runs the media pipeline for a claimed video.
- Pipeline → Detector: run inference · Invokes detection over extracted frames.
- Settings → Pipeline: configure · Provides runtime settings for orchestration and worker behavior.
- Settings → Scraper: configure · Supplies search and discovery settings.
- Keywords → Scraper: seed search · Feeds search terms and URLs into discovery.
- Keyword Gen → Keywords: generate · Produces combinatorial search terms for scraper input.
- Scraper → YouTube: search · Discovers candidate videos via YouTube InnerTube search.
- Scraper → Queue: enqueue · Persists discovered videos as durable work items.
- Queue → Pipeline: claim · Workers atomically claim pending rows using skip-locked semantics.
- Queue → Downloader: dequeue · Provides the next video to process.
- Downloader → YouTube: fetch · Downloads the selected source video.
- Downloader → FFmpeg: mux · Uses FFmpeg when stream muxing is required.
- Downloader → Extractor: hand off · Passes downloaded video files to frame extraction.
- Extractor → FFmpeg: slice frames · Extracts frames with FFmpeg subprocesses.
- Extractor → Detector: frames · Sends extracted frames to the inference stage.
- Detector → YOLOv8: infer · Runs batch YOLOv8 inference on frames.
- Detector → Weights: load · Consumes the trained detection weights.
- Detector → Database: persist results · Writes frame hits and bounding-box detections.
- Database → Videos: store metadata · Maintains video-level records.
- Database → Frame hits: store hits · Stores frames with at least one detection.
- Database → Detections: store boxes · Stores individual detection boxes.
- Database → Queue: state updates · Tracks queue lifecycle transitions.
- Appearances → Database: query · Reads detections for appearance-range analytics.