How akilanpl/survey-data-intelligence works
SIRL turns survey submissions into normalized, profiled batches, then runs deterministic rules, statistics, and ML validation before fusing evidence into risk assessments and classifications. Optional AI explains selected assessments but cannot affect scoring. Supervisors review results in the dashboard and manage investigations. The README documents the end-to-end stages; sampled code confirms selected engine, dashboard, explanation, and investigation behaviors, so uncertain internal wiring is omitted.
Main entry points: orchestrator.py Pipeline orchestration · main.py API application · db.py SQLite state · parquet.py Batch Parquet
Components 21
Supervisor interface
- Dashboard UI
frontend/app/dashboard - Authentication
backend/app/modules/auth - Dashboard queries
backend/app/modules/dashboard/service.py
Ingestion and pipeline
- Input standardization
backend/app/modules/ingestion - Pipeline orchestration
backend/app/modules/pipeline/orchestrator.py - Batch Parquet
backend/app/modules/storage/parquet.py
Profiling and validation
- SIRL profiling
backend/app/modules/sirl - Rule validation
backend/app/modules/validation/rules/engine.py - Statistical validation
backend/app/modules/validation/statistics/engine.py - ML validation
backend/app/modules/validation/ml/engine.py - Evidence fusion
backend/app/modules/validation/fusion/engine.py - Intelligence analysis
backend/app/modules/validation/intelligence
Explanation and review
- Assessment explanations
backend/app/modules/validation/explanation/service.py - AI provider layer
backend/app/modules/ai - Case investigations
backend/app/modules/investigations/service.py
Platform state
- API application
backend/app/main.py - SQLite state
backend/app/db.py
Other components
- Supervisor
- CSV submission
- eSIGMA source
- AI provider
Connections 30
- Supervisor → Dashboard UI: uses
- CSV submission → Input standardization: submits
- eSIGMA source → Input standardization: provides data
- Dashboard UI → API application: requests
- API application → Authentication: dispatches
- API application → Input standardization: dispatches
- API application → Pipeline orchestration: dispatches
- API application → Dashboard queries: dispatches
- API application → Case investigations: dispatches
- Input standardization → Batch Parquet: writes batch
- Pipeline orchestration → SIRL profiling: runs profiling
- Pipeline orchestration → Rule validation: runs validation
- Pipeline orchestration → Statistical validation: runs validation
- Pipeline orchestration → ML validation: runs validation
- Pipeline orchestration → Evidence fusion: runs fusion
- SIRL profiling → Batch Parquet: reads batch
- Statistical validation → Batch Parquet: reads batch
- ML validation → Batch Parquet: reads batch
- Evidence fusion → Batch Parquet: reads records
- Evidence fusion → Intelligence analysis: classifies
- Rule validation → SQLite state: persists evidence
- Statistical validation → SQLite state: persists evidence
- ML validation → SQLite state: persists evidence
- Evidence fusion → SQLite state: persists assessments
- Assessment explanations → SQLite state: reads assessments
- Assessment explanations → AI provider layer: requests explanation
- AI provider layer → AI provider: calls provider
- Dashboard queries → SQLite state: reads results
- Case investigations → SQLite state: stores cases and audit
- Dashboard UI → Case investigations: manages cases