How shani-singh1/sentient works
1. Overall shape
Sentient is a file-backed geospatial ML pipeline, not a DB-centric app. The core flow is: external geospatial sources are ingested into data/raw, transformed into monthly processed layers in data/processed, assembled into model-ready features in data/features, scored by trained models in models, and then published as outputs in data/results for the API and dashboard.
2. Data sources and ingestion boundary
The ingestion layer in src/ingestion/ is split by source: Sentinel-1, Sentinel-2, Landsat, ERA5, Nightlights/VIIRS, Population/WorldPop, and OSM roads. These modules read from external providers or downloaded source archives and materialize city/month-specific artifacts. The repository’s data/raw/ tree shows the canonical input contract: nested by source, year, month, and city.
3. City- and time-scoped configuration
The pipeline is parameterized by config files in config/, with one main JSON per city and year range. This means the same pipeline logic is reused across Bengaluru, Chennai, Hyderabad, and Mumbai, but the repo’s README and most outputs are centered on Bengaluru as the primary target city. City/month scoping is important for any architecture diagram because every downstream stage preserves that partitioning.
4. Preprocessing and validation
src/preprocessing/ handles raw-data validation and creation of the monthly stress signal. The output of this stage is the normalized, month-aligned geospatial layer set that downstream feature builders consume. This boundary is where raw multi-modal imagery becomes analytically consistent monthly inputs.
5. Feature engineering layer
src/features/ is where the pipeline converts processed geospatial layers into model inputs. build_dataset.py assembles the feature dataset used for training and inference; build_road_risk.py converts city/grid predictions into road-level risk outputs. The artifacts in data/features/ include the training parquet, dataset manifest, and normalization stats, which are part of the reproducibility contract.
6. Model training subsystem
src/training/ contains two training tracks: a primary CNN-temporal pipeline and a tabular fallback pipeline. The model artifacts in models/ show this dual strategy clearly: CNN-temporal weights plus baseline, random forest, gradient boosting, and other trained tabular models. The architecture should show a branching model layer rather than a single monolithic predictor.
7. Inference and scoring
src/inference/ consumes the trained model artifacts and feature datasets to produce risk scores. The main outputs are data/results/risk_scores.parquet, data/results/risk_scores_cnn_temporal.parquet, and data/results/road_risk_ranking.json. In practice, inference is the bridge between model predictions and decision-ready road prioritization.
8. Evaluation and quality gates
src/evaluation/ and src/preprocessing/validate_raw_data.py provide the quality and metrics boundary around the pipeline. data/results/evaluation.json and data/results/raw_data_validation.json indicate that the system preserves both model performance and data validity as first-class outputs, which matters for operational trust.
9. API serving layer
src/api/main.py exposes the scored outputs through FastAPI endpoints such as /metadata, /risk/latest, /risk/ranking, /risk/by_zone, /risk/heatmap, and /risk/roads. This service is read-heavy and file-backed: it serves precomputed artifacts rather than running online model inference per request. For the diagram, place the API as a consumer of data/results and data/features, not of raw inputs.
10. Decision UI layer
src/frontend/app.py is a Streamlit dashboard that visualizes the same risk outputs for operational users. It sits alongside the API as a presentation layer, using the scored artifacts for maps, rankings, and inspection prioritization. The UI is decision-oriented, not an independent data producer.
11. Operational orchestration and reproducibility
The repository uses PowerShell wrappers in scripts/ for pipeline execution, API startup, frontend startup, smoke checks, and inventory generation. Dockerfile and docker-compose.yml provide containerized execution paths, while RUNBOOK.md and docs/ document stage-by-stage operations. This is a batch pipeline plus service stack, not a continuously streaming system.
12. Storage layout as architecture
The data/ tree is the main system boundary: raw source archives, processed monthly files, feature datasets, validation manifests, and final decision outputs all live there. Since there is no database or message bus in the repo, the data filesystem is the integration surface between subsystems. Any accurate diagram should show artifact handoffs via files and directories, with Parquet/NetCDF/TIFF/JSON as the main interchange formats.
13. Key external dependencies
The architecture is anchored on geospatial remote-sensing and environmental sources rather than internal services. The notable external domains are Earth observation imagery, weather reanalysis, nightlights, population rasters, and OpenStreetMap road data. Those sources determine both the shape of the raw data and the temporal alignment strategy used downstream.
14. What the outputs mean
The final product is not just a prediction score; it is a road-level prioritization package. road_risk_ranking.json and related heatmap outputs turn monthly grid-level stress and model scores into actionable maintenance ordering and geometry-aware views for city teams.
Main entry points: pipeline.bengaluru.2020_2024.json City config · raw Raw data · results Results · features Feature store
Components 27
Ingestion
Source-specific acquisition of geospatial and environmental raw artifacts.
- Raw data · Canonical source-aligned inputs partitioned by source, year, month, and city.
data/raw - Sentinel-1 · Downloads or reads SAR composites into city/month raw artifacts.
src/ingestion/sentinel1_ingest.py - Sentinel-2 · Downloads or reads optical composites into city/month raw artifacts.
src/ingestion/sentinel2_ingest.py - Landsat · Builds monthly Landsat composites for downstream preprocessing.
src/ingestion/landsat_ingest.py - ERA5 · Materializes monthly reanalysis climate layers per city.
src/ingestion/era5_ingest.py - Nightlights · Materializes monthly VIIRS nightlight rasters per city.
src/ingestion/nightlights_ingest.py - Population · Materializes WorldPop population rasters as input context.
src/ingestion/population_ingest.py - OSM roads · Downloads road geometry and road-network context from OpenStreetMap.
src/ingestion/osm_ingest.py
Processing
Validation, monthly alignment, and feature assembly stages.
- Raw checks · Validates incoming raw artifacts before monthly processing.
src/preprocessing/validate_raw_data.py - Stress layer · Converts validated raw layers into normalized month-aligned geospatial inputs.
src/preprocessing/monthly_stress.py - Processed data · Normalized per-city monthly parquet layers used by feature builders.
data/processed - Dataset build · Assembles model-ready training and inference datasets.
src/features/build_dataset.py - Road risk · Converts grid or zone scores into road-level risk outputs.
src/features/build_road_risk.py - Feature store · Dataset parquet, manifest, and normalization stats for reproducible modeling.
data/features
Modeling
Training, inference, and evaluation of the risk models.
- CNN temporal · Trains the primary CNN-temporal sequence model.
src/training/train_cnn_temporal.py - Tabular train · Trains fallback tabular models such as RF and gradient boosting.
src/training/train_models.py - Model store · Persisted weights and joblib models used for batch scoring.
models - Risk scoring · Scores feature datasets with trained models to produce risk estimates.
src/inference/score_risk.py - CNN scoring · Scores with the CNN-temporal model and emits dedicated outputs.
src/inference/score_risk_cnn_temporal.py - Evaluation · Computes model quality metrics and validation summaries.
src/evaluation/evaluate.py - Results · Final decision-ready parquet and JSON outputs consumed by services.
data/results
Serving
File-backed API and dashboard consumers of scored outputs.
Operations
Pipeline execution, containers, and documentation.
- City config · Per-city pipeline parameters and date ranges for repeated runs.
config/pipeline.bengaluru.2020_2024.json - Pipeline runner · Wrapper for end-to-end pipeline execution and smoke routines.
scripts/run_pipeline.ps1 - API runner · Wrapper for starting the API service locally or in containers.
scripts/run_api.ps1 - UI runner · Wrapper for starting the dashboard locally or in containers.
scripts/run_frontend.ps1
Connections 40
- City config → Sentinel-1: parameterizes · City and date settings drive source ingestion runs.
- City config → Sentinel-2: parameterizes · City and date settings drive source ingestion runs.
- City config → Landsat: parameterizes · City and date settings drive source ingestion runs.
- City config → ERA5: parameterizes · City and date settings drive source ingestion runs.
- City config → Nightlights: parameterizes · City and date settings drive source ingestion runs.
- City config → Population: parameterizes · City and date settings drive source ingestion runs.
- City config → OSM roads: parameterizes · City and date settings drive source ingestion runs.
- Sentinel-1 → Raw data: writes · Stores Sentinel-1 raw artifacts in the file-backed lake.
- Sentinel-2 → Raw data: writes · Stores Sentinel-2 raw artifacts in the file-backed lake.
- Landsat → Raw data: writes · Stores Landsat composites in the raw tree.
- ERA5 → Raw data: writes · Stores ERA5 monthly netCDF inputs in the raw tree.
- Nightlights → Raw data: writes · Stores VIIRS rasters in the raw tree.
- Population → Raw data: writes · Stores WorldPop rasters in the raw tree.
- OSM roads → Raw data: writes · Stores road geometry snapshots in the raw tree.
- Raw data → Raw checks: feeds · Validated from the canonical raw artifact contract.
- Raw checks → Stress layer: checks · Quality-gated raw inputs flow into monthly preprocessing.
- Stress layer → Processed data: writes · Produces normalized monthly processed layers.
- Processed data → Dataset build: feeds · Processed layers are assembled into model-ready datasets.
- Processed data → Road risk: feeds · Processed grid outputs support road-level postprocessing.
- Dataset build → Feature store: writes · Publishes training and inference datasets plus manifests.
- Feature store → CNN temporal: feeds · Feature datasets are consumed by CNN-temporal training.
- Feature store → Tabular train: feeds · Feature datasets are consumed by tabular training.
- CNN temporal → Model store: writes · Persists deep model weights and training metrics.
- Tabular train → Model store: writes · Persists tabular model artifacts and metrics.
- Model store → CNN scoring: loads · Inference uses the CNN-temporal artifact set.
- Model store → Risk scoring: loads · Inference uses the baseline and tabular artifact set.
- Feature store → CNN scoring: feeds · Inference reads model-ready features from disk.
- Feature store → Risk scoring: feeds · Inference reads model-ready features from disk.
- CNN scoring → Results: writes · Writes CNN-temporal risk outputs.
- Risk scoring → Results: writes · Writes general risk scores and ranking artifacts.
- Road risk → Results: writes · Publishes road-level prioritization outputs.
- Evaluation → Results: writes · Publishes model metrics and validation summaries.
- Results → API: serves · API reads precomputed results for request-time responses.
- Results → Dashboard: serves · Dashboard visualizes scored outputs and rankings.
- Pipeline runner → Sentinel-1: runs · Wrapper entry point for ingestion stage execution.
- Pipeline runner → Stress layer: runs · Wrapper entry point for preprocessing and downstream stages.
- Pipeline runner → CNN temporal: runs · Wrapper entry point for training execution.
- Pipeline runner → Risk scoring: runs · Wrapper entry point for batch inference.
- API runner → API: starts · Starts the file-backed API service.
- UI runner → Dashboard: starts · Starts the Streamlit dashboard.