# shani-singh1/sentient architecture

> Architecture diagram of the GitHub repository shani-singh1/sentient, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shani-singh1/sentient
- Repository: https://github.com/shani-singh1/sentient

## Overview

1. **Overall shape**
   Sentient is a file-backed geospatial ML pipeline, not a DB-centric app. The core flow is: external geospatial sources are ingested into `data/raw`, transformed into monthly processed layers in `data/processed`, assembled into model-ready features in `data/features`, scored by trained models in `models`, and then published as outputs in `data/results` for the API and dashboard.

2. **Data sources and ingestion boundary**
   The ingestion layer in `src/ingestion/` is split by source: Sentinel-1, Sentinel-2, Landsat, ERA5, Nightlights/VIIRS, Population/WorldPop, and OSM roads. These modules read from external providers or downloaded source archives and materialize city/month-specific artifacts. The repository’s `data/raw/` tree shows the canonical input contract: nested by source, year, month, and city.

3. **City- and time-scoped configuration**
   The pipeline is parameterized by config files in `config/`, with one main JSON per city and year range. This means the same pipeline logic is reused across Bengaluru, Chennai, Hyderabad, and Mumbai, but the repo’s README and most outputs are centered on Bengaluru as the primary target city. City/month scoping is important for any architecture diagram because every downstream stage preserves that partitioning.

4. **Preprocessing and validation**
   `src/preprocessing/` handles raw-data validation and creation of the monthly stress signal. The output of this stage is the normalized, month-aligned geospatial layer set that downstream feature builders consume. This boundary is where raw multi-modal imagery becomes analytically consistent monthly inputs.

5. **Feature engineering layer**
   `src/features/` is where the pipeline converts processed geospatial layers into model inputs. `build_dataset.py` assembles the feature dataset used for training and inference; `build_road_risk.py` converts city/grid predictions into road-level risk outputs. The artifacts in `data/features/` include the training parquet, dataset manifest, and normalization stats, which are part of the reproducibility contract.

6. **Model training subsystem**
   `src/training/` contains two training tracks: a primary CNN-temporal pipeline and a tabular fallback pipeline. The model artifacts in `models/` show this dual strategy clearly: CNN-temporal weights plus baseline, random forest, gradient boosting, and other trained tabular models. The architecture should show a branching model layer rather than a single monolithic predictor.

7. **Inference and scoring**
   `src/inference/` consumes the trained model artifacts and feature datasets to produce risk scores. The main outputs are `data/results/risk_scores.parquet`, `data/results/risk_scores_cnn_temporal.parquet`, and `data/results/road_risk_ranking.json`. In practice, inference is the bridge between model predictions and decision-ready road prioritization.

8. **Evaluation and quality gates**
   `src/evaluation/` and `src/preprocessing/validate_raw_data.py` provide the quality and metrics boundary around the pipeline. `data/results/evaluation.json` and `data/results/raw_data_validation.json` indicate that the system preserves both model performance and data validity as first-class outputs, which matters for operational trust.

9. **API serving layer**
   `src/api/main.py` exposes the scored outputs through FastAPI endpoints such as `/metadata`, `/risk/latest`, `/risk/ranking`, `/risk/by_zone`, `/risk/heatmap`, and `/risk/roads`. This service is read-heavy and file-backed: it serves precomputed artifacts rather than running online model inference per request. For the diagram, place the API as a consumer of `data/results` and `data/features`, not of raw inputs.

10. **Decision UI layer**
   `src/frontend/app.py` is a Streamlit dashboard that visualizes the same risk outputs for operational users. It sits alongside the API as a presentation layer, using the scored artifacts for maps, rankings, and inspection prioritization. The UI is decision-oriented, not an independent data producer.

11. **Operational orchestration and reproducibility**
   The repository uses PowerShell wrappers in `scripts/` for pipeline execution, API startup, frontend startup, smoke checks, and inventory generation. `Dockerfile` and `docker-compose.yml` provide containerized execution paths, while `RUNBOOK.md` and `docs/` document stage-by-stage operations. This is a batch pipeline plus service stack, not a continuously streaming system.

12. **Storage layout as architecture**
   The `data/` tree is the main system boundary: raw source archives, processed monthly files, feature datasets, validation manifests, and final decision outputs all live there. Since there is no database or message bus in the repo, the data filesystem is the integration surface between subsystems. Any accurate diagram should show artifact handoffs via files and directories, with Parquet/NetCDF/TIFF/JSON as the main interchange formats.

13. **Key external dependencies**
   The architecture is anchored on geospatial remote-sensing and environmental sources rather than internal services. The notable external domains are Earth observation imagery, weather reanalysis, nightlights, population rasters, and OpenStreetMap road data. Those sources determine both the shape of the raw data and the temporal alignment strategy used downstream.

14. **What the outputs mean**
   The final product is not just a prediction score; it is a road-level prioritization package. `road_risk_ranking.json` and related heatmap outputs turn monthly grid-level stress and model scores into actionable maintenance ordering and geometry-aware views for city teams.

Main entry points: [`config/pipeline.bengaluru.2020_2024.json`](https://github.com/shani-singh1/sentient/blob/main/config/pipeline.bengaluru.2020_2024.json) (City config), [`data/raw`](https://github.com/shani-singh1/sentient/tree/main/data/raw) (Raw data), [`data/results`](https://github.com/shani-singh1/sentient/tree/main/data/results) (Results), [`data/features`](https://github.com/shani-singh1/sentient/tree/main/data/features) (Feature store)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_grp_ingest["Ingestion"]
  node_raw_store[("Raw data<br/>file-backed lake")]
  node_s1_ingest["Sentinel-1<br/>source ingest"]
  node_s2_ingest["Sentinel-2<br/>source ingest"]
  node_landsat_ingest["Landsat<br/>source ingest<br/>[landsat_ingest.py]"]
  node_era5_ingest["ERA5<br/>source ingest<br/>[era5_ingest.py]"]
  node_nightlights_ingest["Nightlights<br/>source ingest"]
  node_population_ingest["Population<br/>source ingest"]
  node_osm_ingest["OSM roads<br/>source ingest<br/>[osm_ingest.py]"]
end

subgraph group_grp_process["Processing"]
  node_raw_validate["Raw checks<br/>validation"]
  node_monthly_stress["Stress layer<br/>monthly preprocessing<br/>[monthly_stress.py]"]
  node_processed_store[("Processed data<br/>monthly layers")]
  node_build_dataset["Dataset build<br/>feature assembly<br/>[build_dataset.py]"]
  node_build_road_risk["Road risk<br/>postprocess<br/>[build_road_risk.py]"]
  node_feature_store[("Feature store<br/>feature artifacts")]
end

subgraph group_grp_model["Modeling"]
  node_train_cnn_temporal["CNN temporal<br/>deep training"]
  node_train_models["Tabular train<br/>baseline training<br/>[train_models.py]"]
  node_model_store[("Model store<br/>trained artifacts")]
  node_score_risk["Risk scoring<br/>batch inference<br/>[score_risk.py]"]
  node_score_cnn_temporal["CNN scoring<br/>batch inference"]
  node_evaluation["Evaluation<br/>metrics gate<br/>[evaluate.py]"]
  node_results_store[("Results<br/>published outputs")]
end

subgraph group_grp_serve["Serving"]
  node_api_main["API<br/>[main.py]"]
  node_frontend_app["Dashboard<br/>Streamlit UI<br/>[app.py]"]
end

subgraph group_grp_ops["Operations"]
  node_config_city["City config<br/>city/year config"]
  node_pipeline_script["Pipeline runner<br/>orchestration script<br/>[run_pipeline.ps1]"]
  node_api_script["API runner<br/>service script<br/>[run_api.ps1]"]
  node_frontend_script["UI runner<br/>service script<br/>[run_frontend.ps1]"]
end

node_config_city -->|"parameterizes"| node_s1_ingest
node_config_city -->|"parameterizes"| node_s2_ingest
node_config_city -->|"parameterizes"| node_landsat_ingest
node_config_city -->|"parameterizes"| node_era5_ingest
node_config_city -->|"parameterizes"| node_nightlights_ingest
node_config_city -->|"parameterizes"| node_population_ingest
node_config_city -->|"parameterizes"| node_osm_ingest
node_s1_ingest -->|"writes"| node_raw_store
node_s2_ingest -->|"writes"| node_raw_store
node_landsat_ingest -->|"writes"| node_raw_store
node_era5_ingest -->|"writes"| node_raw_store
node_nightlights_ingest -->|"writes"| node_raw_store
node_population_ingest -->|"writes"| node_raw_store
node_osm_ingest -->|"writes"| node_raw_store
node_raw_store -->|"feeds"| node_raw_validate
node_raw_validate -->|"checks"| node_monthly_stress
node_monthly_stress -->|"writes"| node_processed_store
node_processed_store -->|"feeds"| node_build_dataset
node_processed_store -->|"feeds"| node_build_road_risk
node_build_dataset -->|"writes"| node_feature_store
node_feature_store -->|"feeds"| node_train_cnn_temporal
node_feature_store -->|"feeds"| node_train_models
node_train_cnn_temporal -->|"writes"| node_model_store
node_train_models -->|"writes"| node_model_store
node_model_store -->|"loads"| node_score_cnn_temporal
node_model_store -->|"loads"| node_score_risk
node_feature_store -->|"feeds"| node_score_cnn_temporal
node_feature_store -->|"feeds"| node_score_risk
node_score_cnn_temporal -->|"writes"| node_results_store
node_score_risk -->|"writes"| node_results_store
node_build_road_risk -->|"writes"| node_results_store
node_evaluation -->|"writes"| node_results_store
node_results_store -->|"serves"| node_api_main
node_results_store -->|"serves"| node_frontend_app
node_pipeline_script -->|"runs"| node_s1_ingest
node_pipeline_script -->|"runs"| node_monthly_stress
node_pipeline_script -->|"runs"| node_train_cnn_temporal
node_pipeline_script -->|"runs"| node_score_risk
node_api_script -->|"starts"| node_api_main
node_frontend_script -->|"starts"| node_frontend_app

click node_config_city "https://github.com/shani-singh1/sentient/blob/main/config/pipeline.bengaluru.2020_2024.json"
click node_raw_store "https://github.com/shani-singh1/sentient/tree/main/data/raw"
click node_s1_ingest "https://github.com/shani-singh1/sentient/blob/main/src/ingestion/sentinel1_ingest.py"
click node_s2_ingest "https://github.com/shani-singh1/sentient/blob/main/src/ingestion/sentinel2_ingest.py"
click node_landsat_ingest "https://github.com/shani-singh1/sentient/blob/main/src/ingestion/landsat_ingest.py"
click node_era5_ingest "https://github.com/shani-singh1/sentient/blob/main/src/ingestion/era5_ingest.py"
click node_nightlights_ingest "https://github.com/shani-singh1/sentient/blob/main/src/ingestion/nightlights_ingest.py"
click node_population_ingest "https://github.com/shani-singh1/sentient/blob/main/src/ingestion/population_ingest.py"
click node_osm_ingest "https://github.com/shani-singh1/sentient/blob/main/src/ingestion/osm_ingest.py"
click node_raw_validate "https://github.com/shani-singh1/sentient/blob/main/src/preprocessing/validate_raw_data.py"
click node_monthly_stress "https://github.com/shani-singh1/sentient/blob/main/src/preprocessing/monthly_stress.py"
click node_processed_store "https://github.com/shani-singh1/sentient/tree/main/data/processed"
click node_build_dataset "https://github.com/shani-singh1/sentient/blob/main/src/features/build_dataset.py"
click node_build_road_risk "https://github.com/shani-singh1/sentient/blob/main/src/features/build_road_risk.py"
click node_feature_store "https://github.com/shani-singh1/sentient/tree/main/data/features"
click node_train_cnn_temporal "https://github.com/shani-singh1/sentient/blob/main/src/training/train_cnn_temporal.py"
click node_train_models "https://github.com/shani-singh1/sentient/blob/main/src/training/train_models.py"
click node_model_store "https://github.com/shani-singh1/sentient/tree/main/models"
click node_score_risk "https://github.com/shani-singh1/sentient/blob/main/src/inference/score_risk.py"
click node_score_cnn_temporal "https://github.com/shani-singh1/sentient/blob/main/src/inference/score_risk_cnn_temporal.py"
click node_evaluation "https://github.com/shani-singh1/sentient/blob/main/src/evaluation/evaluate.py"
click node_results_store "https://github.com/shani-singh1/sentient/tree/main/data/results"
click node_api_main "https://github.com/shani-singh1/sentient/blob/main/src/api/main.py"
click node_frontend_app "https://github.com/shani-singh1/sentient/blob/main/src/frontend/app.py"
click node_pipeline_script "https://github.com/shani-singh1/sentient/blob/main/scripts/run_pipeline.ps1"
click node_api_script "https://github.com/shani-singh1/sentient/blob/main/scripts/run_api.ps1"
click node_frontend_script "https://github.com/shani-singh1/sentient/blob/main/scripts/run_frontend.ps1"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_raw_store,node_s1_ingest,node_s2_ingest,node_landsat_ingest,node_era5_ingest,node_nightlights_ingest,node_population_ingest,node_osm_ingest toneBlue
class node_raw_validate,node_monthly_stress,node_processed_store,node_build_dataset,node_build_road_risk,node_feature_store toneAmber
class node_train_cnn_temporal,node_train_models,node_model_store,node_score_risk,node_score_cnn_temporal,node_evaluation,node_results_store toneMint
class node_api_main,node_frontend_app toneRose
class node_config_city,node_pipeline_script,node_api_script,node_frontend_script toneIndigo
```

## Components

### Ingestion

Source-specific acquisition of geospatial and environmental raw artifacts.

- [Raw data](https://github.com/shani-singh1/sentient/tree/main/data/raw): file-backed lake · Canonical source-aligned inputs partitioned by source, year, month, and city. · `data/raw`
- [Sentinel-1](https://github.com/shani-singh1/sentient/blob/main/src/ingestion/sentinel1_ingest.py): source ingest · Downloads or reads SAR composites into city/month raw artifacts. · `src/ingestion/sentinel1_ingest.py`
- [Sentinel-2](https://github.com/shani-singh1/sentient/blob/main/src/ingestion/sentinel2_ingest.py): source ingest · Downloads or reads optical composites into city/month raw artifacts. · `src/ingestion/sentinel2_ingest.py`
- [Landsat](https://github.com/shani-singh1/sentient/blob/main/src/ingestion/landsat_ingest.py): source ingest · Builds monthly Landsat composites for downstream preprocessing. · `src/ingestion/landsat_ingest.py`
- [ERA5](https://github.com/shani-singh1/sentient/blob/main/src/ingestion/era5_ingest.py): source ingest · Materializes monthly reanalysis climate layers per city. · `src/ingestion/era5_ingest.py`
- [Nightlights](https://github.com/shani-singh1/sentient/blob/main/src/ingestion/nightlights_ingest.py): source ingest · Materializes monthly VIIRS nightlight rasters per city. · `src/ingestion/nightlights_ingest.py`
- [Population](https://github.com/shani-singh1/sentient/blob/main/src/ingestion/population_ingest.py): source ingest · Materializes WorldPop population rasters as input context. · `src/ingestion/population_ingest.py`
- [OSM roads](https://github.com/shani-singh1/sentient/blob/main/src/ingestion/osm_ingest.py): source ingest · Downloads road geometry and road-network context from OpenStreetMap. · `src/ingestion/osm_ingest.py`

### Processing

Validation, monthly alignment, and feature assembly stages.

- [Raw checks](https://github.com/shani-singh1/sentient/blob/main/src/preprocessing/validate_raw_data.py): validation · Validates incoming raw artifacts before monthly processing. · `src/preprocessing/validate_raw_data.py`
- [Stress layer](https://github.com/shani-singh1/sentient/blob/main/src/preprocessing/monthly_stress.py): monthly preprocessing · Converts validated raw layers into normalized month-aligned geospatial inputs. · `src/preprocessing/monthly_stress.py`
- [Processed data](https://github.com/shani-singh1/sentient/tree/main/data/processed): monthly layers · Normalized per-city monthly parquet layers used by feature builders. · `data/processed`
- [Dataset build](https://github.com/shani-singh1/sentient/blob/main/src/features/build_dataset.py): feature assembly · Assembles model-ready training and inference datasets. · `src/features/build_dataset.py`
- [Road risk](https://github.com/shani-singh1/sentient/blob/main/src/features/build_road_risk.py): postprocess · Converts grid or zone scores into road-level risk outputs. · `src/features/build_road_risk.py`
- [Feature store](https://github.com/shani-singh1/sentient/tree/main/data/features): feature artifacts · Dataset parquet, manifest, and normalization stats for reproducible modeling. · `data/features`

### Modeling

Training, inference, and evaluation of the risk models.

- [CNN temporal](https://github.com/shani-singh1/sentient/blob/main/src/training/train_cnn_temporal.py): deep training · Trains the primary CNN-temporal sequence model. · `src/training/train_cnn_temporal.py`
- [Tabular train](https://github.com/shani-singh1/sentient/blob/main/src/training/train_models.py): baseline training · Trains fallback tabular models such as RF and gradient boosting. · `src/training/train_models.py`
- [Model store](https://github.com/shani-singh1/sentient/tree/main/models): trained artifacts · Persisted weights and joblib models used for batch scoring. · `models`
- [Risk scoring](https://github.com/shani-singh1/sentient/blob/main/src/inference/score_risk.py): batch inference · Scores feature datasets with trained models to produce risk estimates. · `src/inference/score_risk.py`
- [CNN scoring](https://github.com/shani-singh1/sentient/blob/main/src/inference/score_risk_cnn_temporal.py): batch inference · Scores with the CNN-temporal model and emits dedicated outputs. · `src/inference/score_risk_cnn_temporal.py`
- [Evaluation](https://github.com/shani-singh1/sentient/blob/main/src/evaluation/evaluate.py): metrics gate · Computes model quality metrics and validation summaries. · `src/evaluation/evaluate.py`
- [Results](https://github.com/shani-singh1/sentient/tree/main/data/results): published outputs · Final decision-ready parquet and JSON outputs consumed by services. · `data/results`

### Serving

File-backed API and dashboard consumers of scored outputs.

- [API](https://github.com/shani-singh1/sentient/blob/main/src/api/main.py): FastAPI service · Read-only service exposing precomputed risk artifacts. · `src/api/main.py`
- [Dashboard](https://github.com/shani-singh1/sentient/blob/main/src/frontend/app.py): Streamlit UI · Operational dashboard for maps, rankings, and inspection views. · `src/frontend/app.py`

### Operations

Pipeline execution, containers, and documentation.

- [City config](https://github.com/shani-singh1/sentient/blob/main/config/pipeline.bengaluru.2020_2024.json): city/year config · Per-city pipeline parameters and date ranges for repeated runs. · `config/pipeline.bengaluru.2020_2024.json`
- [Pipeline runner](https://github.com/shani-singh1/sentient/blob/main/scripts/run_pipeline.ps1): orchestration script · Wrapper for end-to-end pipeline execution and smoke routines. · `scripts/run_pipeline.ps1`
- [API runner](https://github.com/shani-singh1/sentient/blob/main/scripts/run_api.ps1): service script · Wrapper for starting the API service locally or in containers. · `scripts/run_api.ps1`
- [UI runner](https://github.com/shani-singh1/sentient/blob/main/scripts/run_frontend.ps1): service script · Wrapper for starting the dashboard locally or in containers. · `scripts/run_frontend.ps1`

## Connections

- City config → Sentinel-1: parameterizes · City and date settings drive source ingestion runs.
- City config → Sentinel-2: parameterizes · City and date settings drive source ingestion runs.
- City config → Landsat: parameterizes · City and date settings drive source ingestion runs.
- City config → ERA5: parameterizes · City and date settings drive source ingestion runs.
- City config → Nightlights: parameterizes · City and date settings drive source ingestion runs.
- City config → Population: parameterizes · City and date settings drive source ingestion runs.
- City config → OSM roads: parameterizes · City and date settings drive source ingestion runs.
- Sentinel-1 → Raw data: writes · Stores Sentinel-1 raw artifacts in the file-backed lake.
- Sentinel-2 → Raw data: writes · Stores Sentinel-2 raw artifacts in the file-backed lake.
- Landsat → Raw data: writes · Stores Landsat composites in the raw tree.
- ERA5 → Raw data: writes · Stores ERA5 monthly netCDF inputs in the raw tree.
- Nightlights → Raw data: writes · Stores VIIRS rasters in the raw tree.
- Population → Raw data: writes · Stores WorldPop rasters in the raw tree.
- OSM roads → Raw data: writes · Stores road geometry snapshots in the raw tree.
- Raw data → Raw checks: feeds · Validated from the canonical raw artifact contract.
- Raw checks → Stress layer: checks · Quality-gated raw inputs flow into monthly preprocessing.
- Stress layer → Processed data: writes · Produces normalized monthly processed layers.
- Processed data → Dataset build: feeds · Processed layers are assembled into model-ready datasets.
- Processed data → Road risk: feeds · Processed grid outputs support road-level postprocessing.
- Dataset build → Feature store: writes · Publishes training and inference datasets plus manifests.
- Feature store → CNN temporal: feeds · Feature datasets are consumed by CNN-temporal training.
- Feature store → Tabular train: feeds · Feature datasets are consumed by tabular training.
- CNN temporal → Model store: writes · Persists deep model weights and training metrics.
- Tabular train → Model store: writes · Persists tabular model artifacts and metrics.
- Model store → CNN scoring: loads · Inference uses the CNN-temporal artifact set.
- Model store → Risk scoring: loads · Inference uses the baseline and tabular artifact set.
- Feature store → CNN scoring: feeds · Inference reads model-ready features from disk.
- Feature store → Risk scoring: feeds · Inference reads model-ready features from disk.
- CNN scoring → Results: writes · Writes CNN-temporal risk outputs.
- Risk scoring → Results: writes · Writes general risk scores and ranking artifacts.
- Road risk → Results: writes · Publishes road-level prioritization outputs.
- Evaluation → Results: writes · Publishes model metrics and validation summaries.
- Results → API: serves · API reads precomputed results for request-time responses.
- Results → Dashboard: serves · Dashboard visualizes scored outputs and rankings.
- Pipeline runner → Sentinel-1: runs · Wrapper entry point for ingestion stage execution.
- Pipeline runner → Stress layer: runs · Wrapper entry point for preprocessing and downstream stages.
- Pipeline runner → CNN temporal: runs · Wrapper entry point for training execution.
- Pipeline runner → Risk scoring: runs · Wrapper entry point for batch inference.
- API runner → API: starts · Starts the file-backed API service.
- UI runner → Dashboard: starts · Starts the Streamlit dashboard.