# sfbrigade/datasci-firerisk architecture

> Architecture diagram of the GitHub repository sfbrigade/datasci-firerisk, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sfbrigade/datasci-firerisk
- Repository: https://github.com/sfbrigade/datasci-firerisk

## Overview

##### Project type and purpose
- This is a data-science–driven predictive modeling and visualization project (“San Francisco Fire Risk Project”) that ingests heterogeneous data sources, builds a machine-learning model to estimate relative fire risk, and produces an interactive map/dashboard as the user-facing interface.
- Its primary features include data ingestion/cleaning, feature engineering (address matching, distance to station, tax data aggregation, complaint features), model training & scoring, and map generation for visualization.

##### File structure
- Top-level scripts & notebooks:
- process.py, final_dataset_upload.py, column_report.py, deduplicate_model_output.py, multiclass.py (batch-oriented ETL, training, reporting)
- Jupyter notebooks (.ipynb) and R scripts for exploratory data analysis (EDA)
- GIS utility: gis.py
- map_template.html / map.html for final visualization
- package.json + lib/*.js for JS-based merging, aggregation, prediction tasks
- Data directories:
- addresses/ (raw address ZIP)
- address_matching/ (Python module to match addresses)
- nearest_fire_station/ (distance calculation to stations, Google API wrappers)
- External config/build: .gitignore, .idea, README.md
- No monolithic web-server; visualization is generated as static HTML (CartoDB embeds, Leaflet, etc.), but could be wrapped in a simple Node/Express host if desired.

##### Extract architecture insights from README
- Data sources: SF OpenData APIs, external Google Distance API, shapefiles, tax & complaint data
- Tech stack: Python (NumPy, pandas, scikit-learn, matplotlib), R (dplyr, ggplot2), Jupyter, JavaScript (for light ETL), CartoDB for map hosting

##### Final reminders
- Be exhaustive: represent each directory or script as a component or sub-component.
- Batch vs. interactive pathways.
- Config or parameter files if they affect the flow.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Data Sources"
        DS1["SF OpenData APIs"]:::data
        DS2["addresses ZIP"]:::data
        DS3["Fire Station CSV & Distance Data"]:::data
        DS4["Google Distance Matrix API"]:::external
        DS5["Static Files (shapefiles, tax CSVs)"]:::data
    end

    subgraph "Data Ingestion & Storage"
        IM1["process.py"]:::etl
        IM2["final_dataset_upload.py"]:::etl
        IM3["match_eas.py"]:::etl
        AM1["address_matching Module"]:::etl
        DSvc1["Distance-to-Station Service"]:::etl
    end

    subgraph "ETL & Feature Engineering"
        JS1["aggregate-tax-data.js"]:::etl
        JS2["merge-fire-data-by-address.js"]:::etl
        RPT1["column_report.py"]:::etl
        RPT2["deduplicate_model_output.py"]:::etl
    end

    subgraph "Modeling"
        M1["multiclass.py"]:::model
        NB["Jupyter Notebooks"]:::model
    end

    subgraph "Visualization & Serving"
        V1["gis.py"]:::viz
        V2["map_template.html"]:::viz
        V3["map.html"]:::viz
        SV["Node/Express Server (Optional)"]:::viz
    end

    subgraph "External Services"
        ES1["CartoDB Hosting"]:::external
    end

    DS1 --> IM1
    DS2 --> IM1
    DS3 --> IM1
    DS4 --> DSvc1
    DSvc1 --> IM1
    IM1 --> AM1
    IM1 --> IM2
    IM1 --> IM3
    AM1 --> JS1
    IM1 --> JS1
    JS1 --> JS2
    JS2 --> RPT1
    JS2 --> RPT2
    RPT2 --> M1
    IM1 --> NB
    NB --> M1
    M1 --> V1
    V1 --> V2
    V2 --> V3
    V3 --> ES1
    SV --> V3

    click DS2 "https://github.com/sfbrigade/datasci-firerisk/blob/master/addresses/SF-FireRisk_Enterprise-Addressing-System-Addresses.zip"
    click DS3 "https://github.com/sfbrigade/datasci-firerisk/blob/master/nearest_fire_station/fire_stations_20171101_andirs.csv"
    click DS3 "https://github.com/sfbrigade/datasci-firerisk/blob/master/nearest_fire_station/nearest_fire_station_20171101_andirs.csv"
    click DS3 "https://github.com/sfbrigade/datasci-firerisk/blob/master/nearest_fire_station/waypoint_list.pickle"
    click IM1 "https://github.com/sfbrigade/datasci-firerisk/blob/master/process.py"
    click IM2 "https://github.com/sfbrigade/datasci-firerisk/blob/master/final_dataset_upload.py"
    click IM3 "https://github.com/sfbrigade/datasci-firerisk/blob/master/match_eas.py"
    click AM1 "https://github.com/sfbrigade/datasci-firerisk/blob/master/address_matching/__init__.py"
    click AM1 "https://github.com/sfbrigade/datasci-firerisk/blob/master/address_matching/addressqry.py"
    click DSvc1 "https://github.com/sfbrigade/datasci-firerisk/blob/master/nearest_fire_station/retrieve_dridistance_from_api.py"
    click DSvc1 "https://github.com/sfbrigade/datasci-firerisk/blob/master/nearest_fire_station/get_nearest_station.py"
    click JS1 "https://github.com/sfbrigade/datasci-firerisk/blob/master/lib/aggregate-tax-data.js"
    click JS2 "https://github.com/sfbrigade/datasci-firerisk/blob/master/lib/merge-fire-data-by-address.js"
    click RPT1 "https://github.com/sfbrigade/datasci-firerisk/blob/master/column_report.py"
    click RPT2 "https://github.com/sfbrigade/datasci-firerisk/blob/master/deduplicate_model_output.py"
    click M1 "https://github.com/sfbrigade/datasci-firerisk/blob/master/multiclass.py"
    click NB "https://github.com/sfbrigade/datasci-firerisk/blob/master/Fire Safety EDA.ipynb"
    click NB "https://github.com/sfbrigade/datasci-firerisk/blob/master/exploratory_data_analysis/EDA Fire Incidents.ipynb"
    click NB "https://github.com/sfbrigade/datasci-firerisk/blob/master/exploratory_data_analysis/Fire complaint exploratory.ipynb"
    click NB "https://github.com/sfbrigade/datasci-firerisk/blob/master/incidents-with-census-blocks.ipynb"
    click V1 "https://github.com/sfbrigade/datasci-firerisk/blob/master/gis.py"
    click V2 "https://github.com/sfbrigade/datasci-firerisk/blob/master/map_template.html"
    click V3 "https://github.com/sfbrigade/datasci-firerisk/blob/master/map.html"
    click SV "https://github.com/sfbrigade/datasci-firerisk/blob/master/package.json"
    click SV "https://github.com/sfbrigade/datasci-firerisk/blob/master/lib/predict-fire-risk.js"

    classDef data fill:#cfe2f3,stroke:#0366d6,color:#0366d6
    classDef etl fill:#fdd9b5,stroke:#e36c09,color:#e36c09
    classDef model fill:#d1e7dd,stroke:#198754,color:#198754
    classDef viz fill:#e2d9f3,stroke:#6f42c1,color:#6f42c1
    classDef external fill:#fff3cd,stroke-dasharray: 5 5,stroke:#d39e00,color:#d39e00
```
