# kfrural/safestreet-pipeline architecture

> Architecture diagram of the GitHub repository kfrural/safestreet-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kfrural/safestreet-pipeline
- Repository: https://github.com/kfrural/safestreet-pipeline

## Overview

SafeStreet is a spatial intelligence platform for analyzing nighttime crime against urban infrastructure. Its primary workflow ingests crime and contextual data, assigns records to H3 cells, computes spatial vulnerability measures, persists results in PostGIS, and presents analysis through a Streamlit dashboard and REST API. The README documents additional socioeconomic, weather, transit, nightlife, events, and sentiment capabilities. Some pipeline and dashboard implementation is unsampled, so those broader connections are based on README statements; edges supported by excerpts or source-index references cite their evidence.

Main entry points: [`src/pipeline_etl.py`](https://github.com/kfrural/safestreet-pipeline/blob/main/src/pipeline_etl.py) (ETL orchestrator), [`src/dashboard/app.py`](https://github.com/kfrural/safestreet-pipeline/blob/main/src/dashboard/app.py) (Streamlit dashboard), [`src/spatial/postgis_ops.py`](https://github.com/kfrural/safestreet-pipeline/blob/main/src/spatial/postgis_ops.py) (Spatial operations), [`src/db/queries.py`](https://github.com/kfrural/safestreet-pipeline/blob/main/src/db/queries.py) (Data queries)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Data ingestion"]
  node_ssp["SSP-SP records<br/>[ssp_sp.py]"]
  node_crimeprep["Crime preparation<br/>[preprocessing.py]"]
  node_infra["OSM infrastructure<br/>[osm_infra.py]"]
  node_light["Official lighting<br/>[lighting.py]"]
  node_etl["ETL orchestrator<br/>[pipeline_etl.py]"]
end

subgraph group_analysis["Spatial analysis"]
  node_h3["H3 indexing<br/>[h3_indexer.py]"]
  node_postgisops["Spatial operations<br/>[postgis_ops.py]"]
  node_spatialstats["Spatial dependence<br/>[spatial.py]"]
  node_statistics["Statistical models<br/>[statistics.py]"]
  node_insights["Insights and confidence<br/>[insights.py]"]
end

subgraph group_persistence["Database"]
  node_db[("PostGIS database<br/>[models.py]")]
  node_queries["Data queries<br/>[queries.py]"]
  node_connection["Database connection<br/>[connection.py]"]
  node_cache["Cached queries<br/>[cached_queries.py]"]
end

subgraph group_interfaces["User interfaces"]
  node_dashboard["Streamlit dashboard<br/>[app.py]"]
  node_map["Interactive maps<br/>[map_viz.py]"]
  node_api["REST API<br/>[app.py]"]
end

subgraph group_context["Context enrichment"]
  node_enrichment["Context sources"]
  node_ibge["IBGE indicators<br/>[ibge.py]"]
  node_weather["Historical weather<br/>[weather.py]"]
  node_nightlife["Nighttime context<br/>[transport_night.py]"]
  node_sentiment["News sentiment<br/>[sentiment.py]"]
end

node_analyst(("Analyst"))
node_operators(("Pipeline operator"))
node_external{{"External data services"}}

node_operators -->|"runs pipeline"| node_etl
node_etl -->|"ingests records"| node_ssp
node_etl -->|"prepares data"| node_crimeprep
node_etl -->|"fetches infrastructure"| node_infra
node_etl -->|"fetches lighting"| node_light
node_etl -->|"fetches indicators"| node_ibge
node_etl -->|"fetches weather"| node_weather
node_etl -.->|"enriches transit"| node_nightlife
node_etl -.->|"analyzes headlines"| node_sentiment
node_external -->|"supplies crime data"| node_ssp
node_external -->|"supplies OSM data"| node_infra
node_external -->|"supplies lighting"| node_light
node_external -->|"supplies indicators"| node_ibge
node_external -->|"supplies weather"| node_weather
node_external -->|"supplies news feeds"| node_sentiment
node_etl -->|"indexes geometries"| node_h3
node_etl -->|"stores records"| node_db
node_etl -->|"runs spatial analysis"| node_postgisops
node_postgisops -->|"builds spatial weights"| node_spatialstats
node_postgisops -->|"uses analytics"| node_statistics
node_postgisops -->|"updates H3 metrics"| node_db
node_queries -->|"opens connection"| node_connection
node_connection -->|"connects to"| node_db
node_cache -->|"wraps queries"| node_queries
node_analyst -->|"explores analysis"| node_dashboard
node_dashboard -->|"loads cached data"| node_cache
node_dashboard -->|"renders map"| node_map
node_dashboard -->|"presents statistics"| node_statistics
node_dashboard -->|"presents insights"| node_insights
node_analyst -->|"requests data"| node_api
node_api -->|"reads results"| node_queries

click node_ssp "https://github.com/kfrural/safestreet-pipeline/blob/main/src/data/ssp_sp.py"
click node_crimeprep "https://github.com/kfrural/safestreet-pipeline/blob/main/src/data/preprocessing.py"
click node_infra "https://github.com/kfrural/safestreet-pipeline/blob/main/src/data/osm_infra.py"
click node_light "https://github.com/kfrural/safestreet-pipeline/blob/main/src/data/lighting.py"
click node_enrichment "https://github.com/kfrural/safestreet-pipeline/tree/main/src/data"
click node_ibge "https://github.com/kfrural/safestreet-pipeline/blob/main/src/data/ibge.py"
click node_weather "https://github.com/kfrural/safestreet-pipeline/blob/main/src/data/weather.py"
click node_nightlife "https://github.com/kfrural/safestreet-pipeline/blob/main/src/data/transport_night.py"
click node_sentiment "https://github.com/kfrural/safestreet-pipeline/blob/main/src/data/sentiment.py"
click node_etl "https://github.com/kfrural/safestreet-pipeline/blob/main/src/pipeline_etl.py"
click node_h3 "https://github.com/kfrural/safestreet-pipeline/blob/main/src/spatial/h3_indexer.py"
click node_postgisops "https://github.com/kfrural/safestreet-pipeline/blob/main/src/spatial/postgis_ops.py"
click node_spatialstats "https://github.com/kfrural/safestreet-pipeline/blob/main/src/analytics/spatial.py"
click node_statistics "https://github.com/kfrural/safestreet-pipeline/blob/main/src/analytics/statistics.py"
click node_insights "https://github.com/kfrural/safestreet-pipeline/blob/main/src/analytics/insights.py"
click node_db "https://github.com/kfrural/safestreet-pipeline/blob/main/src/db/models.py"
click node_queries "https://github.com/kfrural/safestreet-pipeline/blob/main/src/db/queries.py"
click node_connection "https://github.com/kfrural/safestreet-pipeline/blob/main/src/db/connection.py"
click node_cache "https://github.com/kfrural/safestreet-pipeline/blob/main/src/db/cached_queries.py"
click node_dashboard "https://github.com/kfrural/safestreet-pipeline/blob/main/src/dashboard/app.py"
click node_map "https://github.com/kfrural/safestreet-pipeline/blob/main/src/dashboard/components/map_viz.py"
click node_api "https://github.com/kfrural/safestreet-pipeline/blob/main/src/api/app.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ssp,node_crimeprep,node_infra,node_light,node_etl toneBlue
class node_h3,node_postgisops,node_spatialstats,node_statistics,node_insights toneAmber
class node_db,node_queries,node_connection,node_cache toneMint
class node_dashboard,node_map,node_api toneRose
class node_enrichment,node_ibge,node_weather,node_nightlife,node_sentiment,node_analyst,node_operators,node_external toneIndigo
```

## Components

### Data ingestion

- [SSP-SP records](https://github.com/kfrural/safestreet-pipeline/blob/main/src/data/ssp_sp.py): `src/data/ssp_sp.py`
- [Crime preparation](https://github.com/kfrural/safestreet-pipeline/blob/main/src/data/preprocessing.py): `src/data/preprocessing.py`
- [OSM infrastructure](https://github.com/kfrural/safestreet-pipeline/blob/main/src/data/osm_infra.py): `src/data/osm_infra.py`
- [Official lighting](https://github.com/kfrural/safestreet-pipeline/blob/main/src/data/lighting.py): `src/data/lighting.py`
- [ETL orchestrator](https://github.com/kfrural/safestreet-pipeline/blob/main/src/pipeline_etl.py): `src/pipeline_etl.py`

### Spatial analysis

- [H3 indexing](https://github.com/kfrural/safestreet-pipeline/blob/main/src/spatial/h3_indexer.py): `src/spatial/h3_indexer.py`
- [Spatial operations](https://github.com/kfrural/safestreet-pipeline/blob/main/src/spatial/postgis_ops.py): `src/spatial/postgis_ops.py`
- [Spatial dependence](https://github.com/kfrural/safestreet-pipeline/blob/main/src/analytics/spatial.py): `src/analytics/spatial.py`
- [Statistical models](https://github.com/kfrural/safestreet-pipeline/blob/main/src/analytics/statistics.py): `src/analytics/statistics.py`
- [Insights and confidence](https://github.com/kfrural/safestreet-pipeline/blob/main/src/analytics/insights.py): `src/analytics/insights.py`

### Database

- [PostGIS database](https://github.com/kfrural/safestreet-pipeline/blob/main/src/db/models.py): `src/db/models.py`
- [Data queries](https://github.com/kfrural/safestreet-pipeline/blob/main/src/db/queries.py): `src/db/queries.py`
- [Database connection](https://github.com/kfrural/safestreet-pipeline/blob/main/src/db/connection.py): `src/db/connection.py`
- [Cached queries](https://github.com/kfrural/safestreet-pipeline/blob/main/src/db/cached_queries.py): `src/db/cached_queries.py`

### User interfaces

- [Streamlit dashboard](https://github.com/kfrural/safestreet-pipeline/blob/main/src/dashboard/app.py): `src/dashboard/app.py`
- [Interactive maps](https://github.com/kfrural/safestreet-pipeline/blob/main/src/dashboard/components/map_viz.py): `src/dashboard/components/map_viz.py`
- [REST API](https://github.com/kfrural/safestreet-pipeline/blob/main/src/api/app.py): `src/api/app.py`

### Context enrichment

- [Context sources](https://github.com/kfrural/safestreet-pipeline/tree/main/src/data): `src/data`
- [IBGE indicators](https://github.com/kfrural/safestreet-pipeline/blob/main/src/data/ibge.py): `src/data/ibge.py`
- [Historical weather](https://github.com/kfrural/safestreet-pipeline/blob/main/src/data/weather.py): `src/data/weather.py`
- [Nighttime context](https://github.com/kfrural/safestreet-pipeline/blob/main/src/data/transport_night.py): `src/data/transport_night.py`
- [News sentiment](https://github.com/kfrural/safestreet-pipeline/blob/main/src/data/sentiment.py): `src/data/sentiment.py`

### Other components

- **Analyst**
- **Pipeline operator**
- **External data services**

## Connections

- Pipeline operator → ETL orchestrator: runs pipeline
- ETL orchestrator → SSP-SP records: ingests records (evidence: `src/pipeline_etl.py`)
- ETL orchestrator → Crime preparation: prepares data (evidence: `src/pipeline_etl.py`)
- ETL orchestrator → OSM infrastructure: fetches infrastructure (evidence: `src/pipeline_etl.py`)
- ETL orchestrator → Official lighting: fetches lighting (evidence: `src/pipeline_etl.py`)
- ETL orchestrator → IBGE indicators: fetches indicators (evidence: `src/pipeline_etl.py`)
- ETL orchestrator → Historical weather: fetches weather (evidence: `src/pipeline_etl.py`)
- ETL orchestrator → Nighttime context: enriches transit
- ETL orchestrator → News sentiment: analyzes headlines (evidence: `src/pipeline_etl.py`)
- External data services → SSP-SP records: supplies crime data
- External data services → OSM infrastructure: supplies OSM data
- External data services → Official lighting: supplies lighting
- External data services → IBGE indicators: supplies indicators
- External data services → Historical weather: supplies weather
- External data services → News sentiment: supplies news feeds
- ETL orchestrator → H3 indexing: indexes geometries (evidence: `src/pipeline_etl.py`)
- ETL orchestrator → PostGIS database: stores records (evidence: `src/pipeline_etl.py`)
- ETL orchestrator → Spatial operations: runs spatial analysis (evidence: `src/pipeline_etl.py`)
- Spatial operations → Spatial dependence: builds spatial weights (evidence: `src/spatial/postgis_ops.py`)
- Spatial operations → Statistical models: uses analytics (evidence: `src/spatial/postgis_ops.py`)
- Spatial operations → PostGIS database: updates H3 metrics (evidence: `src/spatial/postgis_ops.py`)
- Data queries → Database connection: opens connection (evidence: `src/db/queries.py`)
- Database connection → PostGIS database: connects to (evidence: `src/db/connection.py`)
- Cached queries → Data queries: wraps queries (evidence: `src/db/cached_queries.py`)
- Analyst → Streamlit dashboard: explores analysis
- Streamlit dashboard → Cached queries: loads cached data (evidence: `src/dashboard/app.py`)
- Streamlit dashboard → Interactive maps: renders map (evidence: `src/dashboard/app.py`)
- Streamlit dashboard → Statistical models: presents statistics (evidence: `src/dashboard/app.py`)
- Streamlit dashboard → Insights and confidence: presents insights (evidence: `src/dashboard/app.py`)
- Analyst → REST API: requests data
- REST API → Data queries: reads results (evidence: `src/api/app.py`)
