How kfrural/safestreet-pipeline works
SafeStreet is a spatial intelligence platform for analyzing nighttime crime against urban infrastructure. Its primary workflow ingests crime and contextual data, assigns records to H3 cells, computes spatial vulnerability measures, persists results in PostGIS, and presents analysis through a Streamlit dashboard and REST API. The README documents additional socioeconomic, weather, transit, nightlife, events, and sentiment capabilities. Some pipeline and dashboard implementation is unsampled, so those broader connections are based on README statements; edges supported by excerpts or source-index references cite their evidence.
Main entry points: pipeline_etl.py ETL orchestrator · app.py Streamlit dashboard · postgis_ops.py Spatial operations · queries.py Data queries
Components 25
Data ingestion
- SSP-SP records
src/data/ssp_sp.py - Crime preparation
src/data/preprocessing.py - OSM infrastructure
src/data/osm_infra.py - Official lighting
src/data/lighting.py - ETL orchestrator
src/pipeline_etl.py
Spatial analysis
- H3 indexing
src/spatial/h3_indexer.py - Spatial operations
src/spatial/postgis_ops.py - Spatial dependence
src/analytics/spatial.py - Statistical models
src/analytics/statistics.py - Insights and confidence
src/analytics/insights.py
Database
- PostGIS database
src/db/models.py - Data queries
src/db/queries.py - Database connection
src/db/connection.py - Cached queries
src/db/cached_queries.py
User interfaces
- Streamlit dashboard
src/dashboard/app.py - Interactive maps
src/dashboard/components/map_viz.py - REST API
src/api/app.py
Context enrichment
- Context sources
src/data - IBGE indicators
src/data/ibge.py - Historical weather
src/data/weather.py - Nighttime context
src/data/transport_night.py - News sentiment
src/data/sentiment.py
Other components
- Analyst
- Pipeline operator
- External data services
Connections 31
- Pipeline operator → ETL orchestrator: runs pipeline
- ETL orchestrator → SSP-SP records: ingests records
src/pipeline_etl.py - ETL orchestrator → Crime preparation: prepares data
src/pipeline_etl.py - ETL orchestrator → OSM infrastructure: fetches infrastructure
src/pipeline_etl.py - ETL orchestrator → Official lighting: fetches lighting
src/pipeline_etl.py - ETL orchestrator → IBGE indicators: fetches indicators
src/pipeline_etl.py - ETL orchestrator → Historical weather: fetches weather
src/pipeline_etl.py - ETL orchestrator → Nighttime context: enriches transit
- ETL orchestrator → News sentiment: analyzes headlines
src/pipeline_etl.py - External data services → SSP-SP records: supplies crime data
- External data services → OSM infrastructure: supplies OSM data
- External data services → Official lighting: supplies lighting
- External data services → IBGE indicators: supplies indicators
- External data services → Historical weather: supplies weather
- External data services → News sentiment: supplies news feeds
- ETL orchestrator → H3 indexing: indexes geometries
src/pipeline_etl.py - ETL orchestrator → PostGIS database: stores records
src/pipeline_etl.py - ETL orchestrator → Spatial operations: runs spatial analysis
src/pipeline_etl.py - Spatial operations → Spatial dependence: builds spatial weights
src/spatial/postgis_ops.py - Spatial operations → Statistical models: uses analytics
src/spatial/postgis_ops.py - Spatial operations → PostGIS database: updates H3 metrics
src/spatial/postgis_ops.py - Data queries → Database connection: opens connection
src/db/queries.py - Database connection → PostGIS database: connects to
src/db/connection.py - Cached queries → Data queries: wraps queries
src/db/cached_queries.py - Analyst → Streamlit dashboard: explores analysis
- Streamlit dashboard → Cached queries: loads cached data
src/dashboard/app.py - Streamlit dashboard → Interactive maps: renders map
src/dashboard/app.py - Streamlit dashboard → Statistical models: presents statistics
src/dashboard/app.py - Streamlit dashboard → Insights and confidence: presents insights
src/dashboard/app.py - Analyst → REST API: requests data
- REST API → Data queries: reads results
src/api/app.py