How abishek-kk/railmind-ai works
1. Repository shape
- This is a monorepo with two primary deployable applications:
- backend/: a Python 3.11 FastAPI service that owns CV inference, risk scoring, alerting, persistence, and training APIs.
- frontend/: a React 19 single-page dashboard for live monitoring, alerts, and analytics.
- The repo also includes training notebooks/scripts, test suites, and a checked-in SQLite database for local/dev use.
2. End-to-end system boundary
- The architecture is split between an edge processing pipeline and a cloud/API layer.
- Raw CCTV video is processed close to the camera in the backend CV pipeline, converted into compact behavioral signals, then handed to the API/business layer for alerting, storage, and dashboard delivery.
- The frontend never touches video directly; it consumes API data and WebSocket events from the backend.
3. Backend entrypoints and runtime
- backend/app/main.py is the FastAPI application entrypoint; backend/run.py and uvicorn app.main:app are the main launch paths.
- Configuration, DB setup, scheduler, and WebSocket management live under backend/app/core/.
- The backend is organized as a layered service: API routes → service layer → models/database → analytics / agent / CV subsystems.
4. Computer vision and edge inference pipeline
- The CV path is under backend/app/cv/ and backend/app/features/.
- It processes video frames through OpenCV, YOLOv8 person detection, ByteTrack-style tracking, and YOLOv8 pose estimation.
- Feature modules such as edge_proximity, loitering_detector, pacing_detector, following_detector, and movement_analyzer generate the behavioral inputs used downstream.
5. Temporal ML pipeline
- The LSTM subsystem under backend/app/lstm/ builds, preprocesses, trains, and predicts on behavioral sequences.
- The model is designed around sliding temporal windows and is treated as a separate capability from the rule-based feature detectors.
- Training utilities, CLI, and model generation scripts suggest the repo supports both offline training and runtime inference with saved artifacts.
6. Agentic reasoning layer
- backend/app/agents/ contains a three-stage LangGraph-style flow: perception_agent, reasoning_agent, and intervention_agent, composed in agent_graph.py.
- This layer turns model outputs and incident context into risk decisions, escalation levels, and recommended actions.
- It is the main boundary between pure detection and operational response.
7. Business logic and alert lifecycle
- The service layer under backend/app/services/ owns alert creation, escalation, notification, incident handling, training orchestration, and risk scoring.
- risk_scoring.py, alert_service.py, incident_service.py, and escalation_service.py appear to be the core orchestration points for converting detections into actionable incidents.
- Notification delivery is separated from core logic, with support for email/SMS style escalation flows.
8. API surface
- FastAPI routes are grouped by domain in backend/app/api/routes/: alerts, analytics, dashboard, feeds, health, incidents, staff, and training.
- There is a clear split between synchronous REST endpoints for querying/managing state and WebSocket endpoints for live event delivery.
- backend/app/api/deps.py likely provides shared dependencies such as DB sessions, auth/context, or service injection.
9. Persistence and domain model
- Domain entities live under backend/app/models/ with matching request/response schemas under backend/app/schemas/.
- The README indicates core persisted concepts: incidents, alerts, tracks, analytics, staff, platforms, and feedback.
- SQLite is used locally (data/railmind.db), while PostgreSQL is intended for production.
10. Realtime transport
- backend/app/core/websocket_manager.py manages live push delivery to the UI.
- The frontend subscribes to alert and live-feed streams, so the backend acts as the realtime hub for operational monitoring.
- This is important architecturally: the dashboard is not polling for everything; some state changes are event-driven.
11. Analytics and dashboard preparation
- backend/app/analytics/ contains precomputed metrics for heatmaps, incident stats, and dashboard summaries.
- These modules sit between raw incident data and frontend visualization, which implies the dashboard consumes curated aggregates rather than reconstructing analytics client-side.
- This helps keep the frontend thin and the backend authoritative for business metrics.
12. Frontend structure
- frontend/src/routes/ contains the top-level app views: dashboard, alerts, live, and the landing page.
- frontend/src/components/ holds reusable UI pieces such as feed cards, risk badges, stats cards, sidebar/top bar, and a large shadcn/ui component set.
- The frontend is a standard client app with route-based views, shared API clients in frontend/src/lib/api/, and a WebSocket hook in frontend/src/hooks/useWebSocket.ts.
13. Frontend data flow
- The frontend fetches domain data through typed API helper modules (alerts.ts, dashboard.ts, feeds.ts) and subscribes to live updates over WebSocket.
- It is primarily a presentation layer for alerts, feed status, trends, and live risk classification.
- E2E tests under frontend/tests/e2e/ indicate browser-level validation of the live feed and dashboard experience.
14. Training and retraining pipeline
- Training assets live under backend/training/ and backend/app/lstm/.
- There are notebooks for analysis, plus scripts for feature extraction, dataset preparation, evaluation, and LSTM training.
- The backend also exposes training-related API routes, so model training is operationalized rather than being a separate offline-only workflow.
15. Operational tooling and validation
- The repo includes scripts for demo mode, video processing, and stream URL backfilling under backend/scripts/.
- Tests cover agents, CV, LSTM, risk scoring, analytics, alerts, and escalation behavior, which reflects the main architectural seams.
- CI is present in .github/workflows/ci.yml, so backend and frontend behaviors are likely validated automatically.
16. External integrations and boundaries
- Optional integrations include LLM providers for reasoning, SMTP for email escalation, and Twilio for SMS alerts.
- The system is designed to be privacy-preserving: no face recognition, no biometric identity persistence, and track/session IDs are used instead.
- For an architecture diagram, the critical boundaries are: camera/video sources → edge CV/ML → backend reasoning/services/storage → WebSocket/API → React dashboard, with optional notification services branching off the alerting layer.
Main entry points: services Services · api/routes API routes · core Core runtime · src/routes Dashboard views
Components 24
Backend
FastAPI service for CV ingestion, reasoning, persistence, and live delivery.
- FastAPI app · Backend application entrypoint and main runtime bootstrap.
backend/app/main.py - Core runtime · Configuration, database setup, scheduler, and websocket coordination.
backend/app/core - Processor mgr · Coordinates video processors and runtime processing workers.
backend/app/core/processor_manager.py - Realtime hub · Pushes live events and state changes to connected clients.
backend/app/core/websocket_manager.py - CV pipeline · Video processing and pose/detection pipeline that extracts behavior signals.
backend/app/cv - Feature detectors · Rule-based behavioral detectors such as loitering, pacing, following, and proximity.
backend/app/features - LSTM runtime · Sliding-window sequence model for behavior prediction and scoring.
backend/app/lstm - Agent graph · Three-stage perception, reasoning, and intervention flow for operational decisions.
backend/app/agents/agent_graph.py - Services · Alerting, escalation, incident handling, risk scoring, notification, and training orchestration.
backend/app/services - Alerting · Creates and manages alerts and their state transitions.
backend/app/services/alert_service.py - Risk scoring · Converts detections and context into risk levels and priorities.
backend/app/services/risk_scoring.py - Incidents · Builds incidents from detections and manages incident lifecycle.
backend/app/services/incident_service.py - Notifications · Sends outbound notifications such as email/SMS escalation messages.
backend/app/services/notification_service.py - Analytics · Precomputed metrics and summaries for dashboards and trends.
backend/app/analytics - API routes · Domain routes for alerts, analytics, dashboard, feeds, incidents, staff, health, and training.
backend/app/api/routes - Domain model · Persisted entities and request/response schemas for alerts, feeds, incidents, staff, and training runs.
backend/app/models
Frontend
React dashboard for monitoring alerts, feeds, and analytics.
- React app · Frontend application bootstrap and routing setup.
frontend/src/main.tsx - Dashboard views · Top-level pages for dashboard, alerts, live monitoring, and landing.
frontend/src/routes - UI components · Reusable cards, navigation, badges, feed widgets, and shared UI primitives.
frontend/src/components - API client · Typed REST helpers for alerts, dashboard, and feeds.
frontend/src/lib/api - WebSocket hook · Client-side subscription hook for realtime backend events.
frontend/src/hooks/useWebSocket.ts
Training
Offline and operational model training pipeline for temporal behavior models.
- Training assets · Scripts and notebooks for feature extraction, dataset prep, evaluation, and LSTM training.
backend/training
External
Non-repo systems and data sources the application integrates with.
- Camera feeds · CCTV/video streams that enter the edge processing pipeline.
- SQLite DB · Checked-in local database used for development and testing.
data/railmind.db
Connections 34
- Camera feeds → CV pipeline: video frames · Raw CCTV/video streams are processed by the backend CV pipeline.
- FastAPI app → Core runtime: bootstraps · The FastAPI app initializes core runtime services and infrastructure.
- Core runtime → Processor mgr: manages · Core runtime coordinates video processors and workers.
- Core runtime → Realtime hub: hosts · Core runtime exposes websocket delivery infrastructure.
- CV pipeline → Feature detectors: signals · Video processing produces behavioral features for downstream analysis.
- CV pipeline → LSTM runtime: sequences · Temporal behavior sequences are prepared for ML inference.
- Feature detectors → LSTM runtime: features · Detector outputs can be consumed by temporal sequence modeling.
- LSTM runtime → Agent graph: predictions · Model outputs are handed to the agentic reasoning layer.
- Feature detectors → Agent graph: context · Rule-based detections provide context for operational reasoning.
- Agent graph → Risk scoring: decides · Reasoning stage informs risk and escalation decisions.
- Risk scoring → Alerting: raises · Risk results can create or update alerts.
- Risk scoring → Incidents: opens · High-risk events can become incidents.
- Alerting → Notifications: notifies · Alert lifecycle can trigger outbound notifications.
- Incidents → Notifications: escalates · Incident handling can escalate through notification channels.
- Services → Alerting: orchestrates · Central service layer coordinates alert management.
- Services → Risk scoring: orchestrates · Central service layer coordinates risk computation.
- Services → Incidents: orchestrates · Central service layer coordinates incident handling.
- Services → Notifications: orchestrates · Central service layer coordinates external notifications.
- Services → Analytics: feeds · Operational data is summarized into dashboard metrics.
- API routes → Services: calls · REST endpoints delegate to business services.
- API routes → Analytics: serves · Analytics routes expose curated metrics and summaries.
- API routes → Domain model: reads/writes · Endpoints exchange structured domain objects and schemas.
- Domain model → SQLite DB: persists · Domain entities are stored in the SQLite-backed local database.
- Core runtime → SQLite DB: connects · Database setup and sessions are managed from the core runtime.
- Realtime hub → React app: pushes events · Realtime backend events are delivered to the dashboard.
- API routes → React app: serves data · Frontend views query backend REST endpoints for state.
- API client → API routes: fetches · Typed API helpers call the backend route surface.
- WebSocket hook → Realtime hub: subscribes · Client hook listens to backend websocket streams.
- React app → Dashboard views: renders · Frontend app mounts route-based views.
- Dashboard views → UI components: composes · Pages are built from shared UI components.
- Dashboard views → API client: loads data · Route views fetch alerts, dashboard, and feed data.
- Dashboard views → WebSocket hook: listens live · Live views subscribe to realtime updates.
- Training assets → LSTM runtime: trains · Offline scripts and notebooks build and evaluate temporal models.
- API routes → Training assets: exposes training · Training endpoints operationalize model training workflows.