How d3vil13/btp_targeted_enforcement_intelligence works
1. Repository shape
- This is primarily a decision-support platform for Bengaluru traffic enforcement, not a generic CRUD app.
- It has two major runtime surfaces: an offline analytics/modeling pipeline and an online full-stack web app.
- The online app reads precomputed artifacts from the repo root; it does not depend on a live database.
2. Core data source and trust boundary
- The system starts from BTP’s stationary AI edge camera violation dataset.
- Raw records flow through cleaning, deduplication, feature engineering, and human-audited fields.
- Important boundary: the app uses auditor-corrected outputs like updated_vehicle_number and updated_vehicle_type, not raw OCR values.
3. Offline data engineering pipeline
- The notebooks and scripts/ folder form a batch pipeline that turns raw CSV data into structured analysis tables and geospatial assets.
- Key outputs include:
- hotspots_ranked.csv
- station_performance.csv
- daily_violation_counts.csv
- violations_sample.csv
- camera_coverage.geojson, coverage_gaps.geojson, station_boundaries.geojson, hotspot_hulls.geojson
- This layer is the source of truth for almost all dashboard and map content.
4. Spatial intelligence subsystem
- Hotspot detection is based on H3 hexagonal binning and geospatial clustering.
- Camera coverage and station boundary files provide the spatial context for maps and gap analysis.
- The app’s geospatial views are driven by these static GeoJSON artifacts rather than live GIS services.
5. Forecasting subsystem
- Forecast generation is separate from the web app and lives in forecast/ plus models/branch_a/.
- The final forecast artifact is an ensemble combining a day-of-week baseline and a global LSTM, with per-station Prophet models retained as artifacts.
- Outputs are materialized as:
- forecasts_branch_a_station.csv
- forecasts_branch_a.csv
- These feed the forecast page and planning views.
6. Backend service boundary
- webapp/backend is a FastAPI service that exposes the data to the frontend.
- app/services/data_service.py is the main read/compute layer: it loads the precomputed CSV/GeoJSON/model-derived files and serves response-ready data.
- app/api/routes.py defines the API surface consumed by the UI.
- app/models/schemas.py defines the response contracts, so the backend acts as a schema boundary between static analytics data and the UI.
7. Frontend application boundary
- webapp/frontend is a React + TypeScript + Vite SPA.
- It communicates with the backend through src/services/api.ts.
- The frontend is organized by feature pages, not by generic widgets, with pages for Dashboard, Enforcement Map, Priority, Stations, Forecast, Simulator, Analytics, and Settings.
8. Primary user-facing data flows
- Dashboard flow: aggregated counts and summaries from processed CSVs → KPI cards and trend charts.
- Map flow: hotspot rankings + camera coverage + station boundaries + sampled violations → Leaflet visualization.
- Priority flow: hotspot metrics + configurable ROI weights → ranked enforcement actions.
- Station flow: station benchmark metrics → performance table and gap warnings.
- Forecast flow: 7-day predictions → heatmap and station detail chart.
- Simulator flow: forecast/priority data + greedy assignment logic on backend → officer deployment recommendations.
9. Business logic in the backend
- The backend is not just a file server; it contains operational logic for:
- ranking zones by congestion impact,
- computing simulator assignments,
- preparing forecast and station summaries,
- applying tunable weights for priority scoring.
- That makes data_service.py the central orchestration point for derived views.
10. Configuration and deployment model
- The app is intended to be run locally as two independent processes:
- Uvicorn for FastAPI on port 8000
- Vite dev server on port 5173
- The frontend proxies /api requests to the backend, which keeps the browser-facing app decoupled from backend hosting details.
11. Static artifact dependency model
- A notable architectural choice is that the repo includes all generated assets in source control.
- The runtime app depends on local files in the repository root rather than on a database, message queue, or object store.
- This makes the system reproducible and demo-friendly, but also means updates require rerunning the offline pipeline.
12. Optional enrichment path
- scripts/01_fetch_mapmyindia_traffic.py and scripts/02_fetch_osm_roadtype.py can enrich hotspots with external traffic/road-type context.
- These are optional inputs to the analytics pipeline, not required for the core app.
- The architecture therefore supports both a fully offline mode and an enriched mode with external data.
13. Historical/archived subsystems
- _archive/ contains earlier prototypes, older models, notebooks, and generated outputs.
- These are not part of the main runtime path, but they show that the current architecture is the result of iterative experimentation with prior Streamlit, SARIMA, autoencoder, and ensemble approaches.
14. What an architecture diagram should emphasize
- At the center: precomputed analytical artifacts produced by notebooks/scripts.
- On the right: FastAPI backend as the API and data-preparation boundary.
- On the far right: React frontend as the presentation layer.
- On the left/bottom: raw camera violation dataset feeding the batch pipeline.
- Around the batch pipeline: spatial analysis, forecasting, and ROI/deployment optimization as separate derived subsystems.
Main entry points: 01_fetch_mapmyindia_traffic.py Enrichment scripts · data_service.py Data service · routes.py API routes · 05_h3_hotspots.py Hotspot build
Components 20
Offline pipeline
Batch notebooks and scripts that transform raw violation data into static analytical artifacts.
- Camera data · Stationary AI edge-camera violation records; the trusted source feeding the batch pipeline.
- Notebooks · Exploration, cleaning, auditing, and pipeline development notebooks.
dataset_feature_engineering.ipynb - Enrichment scripts · Optional external enrichment and core pipeline steps for map, coverage, hotspot, and forecast preparation.
scripts/01_fetch_mapmyindia_traffic.py - Coverage build · Builds camera coverage and station boundary geospatial layers.
scripts/03_build_camera_coverage.py - Hotspot build · Ranks and clusters violations into hotspot outputs using hex/bin spatial logic.
scripts/05_h3_hotspots.py - Forecast build · Reblends and materializes the branch A forecast artifacts from model components.
forecast/branch_a_ensemble.py
Spatial + forecast assets
Geospatial and prediction outputs materialized in the repository root for downstream consumption.
- Analytic tables · Precomputed summary tables consumed by dashboard, station, priority, and simulator views.
hotspots_ranked.csv - Geo assets · Static geospatial layers for maps, coverage, and gap analysis.
camera_coverage.geojson - Forecasts · 7-day predictions and station-level forecast outputs used by planning views.
forecasts_branch_a.csv - Model store · Persisted LSTM, Prophet, and ensemble-weight artifacts supporting forecast generation.
models/branch_a/lstm.keras - ROI weights · Tunable scoring weights used by priority ranking and deployment logic.
roi_weights.json
FastAPI backend
Read/compute API boundary that loads static artifacts and serves response-ready data.
- Data service · Central orchestration layer that loads artifacts, computes summaries, and prepares API payloads.
webapp/backend/app/services/data_service.py - API routes · FastAPI route definitions consumed by the frontend.
webapp/backend/app/api/routes.py - Schemas · Backend response models defining the boundary between static analytics data and the UI.
webapp/backend/app/models/schemas.py - Backend app · Application bootstrap and server entrypoint for the backend process.
webapp/backend/main.py
React frontend
Vite SPA that renders feature pages and talks to the backend API.
- SPA shell · Vite application shell and top-level routing/layout.
webapp/frontend/src/App.tsx - API client · Frontend service layer for backend communication.
webapp/frontend/src/services/api.ts - Feature pages · Dashboard, map, priority, stations, forecast, simulator, analytics, and settings pages.
webapp/frontend/src/pages/Dashboard.tsx - Shared UI · Reusable layout, cards, tables, and navigation components.
webapp/frontend/src/components/Layout.tsx
Archived work
Older notebooks, models, and prototype apps kept for reference.
- Archive · Older notebooks, models, and prototype apps retained for historical context.
_archive/old_streamlit_app/app/app.py
Connections 21
- Camera data → Notebooks: inspect/clean · Exploration and feature engineering start from the raw violation records.
- Notebooks → Enrichment scripts: derive pipeline · Notebook work is operationalized into batch scripts and reproducible steps.
- Enrichment scripts → Coverage build: build coverage · Spatial preprocessing produces camera coverage and boundary layers.
- Enrichment scripts → Hotspot build: rank hotspots · Violations are cleaned, deduplicated, and aggregated into hotspot metrics.
- Enrichment scripts → Forecast build: prepare forecast · Processed features feed the forecast branch build and reblending steps.
- Hotspot build → Analytic tables: write tables · Hotspot analytics are materialized into ranked and station-level CSV outputs.
- Coverage build → Geo assets: write geojson · Coverage and station geometry outputs are persisted as static map layers.
- Hotspot build → Geo assets: shape maps · Spatial clustering contributes hotspot hulls and map overlays.
- Forecast build → Forecasts: write forecasts · Ensemble outputs are materialized for the forecast page and planning views.
- Model store → Forecast build: serve models · Saved LSTM/Prophet artifacts are inputs to forecast generation.
- ROI weights → Analytic tables: score priority · Priority and deployment rankings use configurable ROI weights.
- Analytic tables → Data service: load summaries · The backend reads summary tables to compute response-ready views.
- Geo assets → Data service: load maps · Static geospatial files are read by the backend for map responses.
- Forecasts → Data service: load forecasts · Forecast payloads are served from precomputed branch A outputs.
- Data service → API routes: serve endpoints · Central service functions back the FastAPI route handlers.
- Schemas → API routes: shape responses · Route handlers return schema-defined payloads to the frontend.
- Backend app → API routes: mount API · The FastAPI app wires the route module into the server process.
- API routes → API client: /api · The SPA consumes backend endpoints through its typed API client.
- API client → Feature pages: fetch view data · Feature pages request dashboard, map, forecast, and simulator data from the backend.
- Feature pages → Shared UI: compose UI · Pages are assembled from shared layout and component primitives.
- Archive → Notebooks: historical context · Archived prototypes show earlier iterations of the same analytical workflow.