# tommella90/italy-house-prices architecture

> Architecture diagram of the GitHub repository tommella90/italy-house-prices, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tommella90/italy-house-prices
- Repository: https://github.com/tommella90/italy-house-prices

## Overview

##### Project type and purpose
- This is a Python-based ETL + interactive data visualization app.
- It scrapes public housing listings from Immobiliare.it, stores and processes them (ETL), then exposes an interactive Streamlit front-end for exploring Italy house prices by city, region, and over time.

##### Main components
a. External Data Source
- Immobiliare.it (HTML pages of listings)
b. Scraping & Ingestion (ETL)
- prepare_data.py – orchestrates HTTP requests, parsing, initial cleaning
- (Implicit scraping logic) → raw CSV/JSON or DataFrame
c. Cloud Storage
- Google Cloud Storage “italy-housing-data” (raw and processed objects)
d. Data Warehouse / Local Data Store
- dataframes/sales.parquet – unified sales dataset
- geodata/… – Excel/CSV/GeoJSON for municipalities, provinces, centroids
e. Data Processing & Enrichment
- affordability.py – computes affordability metrics
- PricesMap.py, maps_italy.py, map_neighbourhoods.py – prepare GeoDataFrames for mapping
f. Interactive Web App (Presentation Layer)
- Streamlit app entrypoint (e.g., playground.py or main script)
- pages/PricesTrends.py – multipage navigation
g. End User
- Browser/UI interacting with Streamlit

##### Relationships & Data Flows
1. Immobiliare.it → prepare_data.py (scrapes raw housing data)
2. prepare_data.py → GCS bucket (uploads raw/clean CSV or Parquet)
3. ETL scripts (prepare_data.py + affordability.py) → dataframes/sales.parquet
4. Geo-assets (geodata/) loaded locally by mapping scripts
5. Streamlit App loads:
- sales.parquet (Pandas/Parquet)
- GeoJSON/CSV/Excel from geodata/
6. Streamlit UI renders maps & charts → User

##### Architectural Patterns & Design Principles
- ETL pipeline → Data Lake → Self-service BI (Streamlit)
- Separation of concerns:
- Ingestion & storage
- Processing & enrichment
- Presentation

##### Must-Include
- Clear labels on every component (script name, file store, service)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Layer
    subgraph "External Services"
        direction TB
        Immobi["Immobiliare.it"]:::external
    end

    %% Ingestion Layer
    subgraph "Ingestion Layer"
        direction TB
        Prepare["prepare_data.py"]:::process
    end

    %% Storage Layer
    subgraph "Storage Layer"
        direction TB
        GCSBucket["GCS Bucket\n(italy-housing-data)"]:::storage
        SalesParquet["sales.parquet"]:::storage
        subgraph "Local Geo Store"
            direction TB
            ComuniExcel["comuni.xlsx"]:::storage
            CoordsCSV["coords.csv"]:::storage
            ProvGeoJSON["prov_coords/*.json"]:::storage
            ItaliaMunic["italy_munic.xlsx"]:::storage
            MunCentroids["municipalities_centroids.csv"]:::storage
            ProvItaliane["province-italiane.xls"]:::storage
            RegCentroids["regions_centroids.csv"]:::storage
        end
    end

    %% Processing Layer
    subgraph "Processing & Enrichment"
        direction TB
        Afford["affordability.py"]:::process
        PriceMap["PricesMap.py"]:::process
        MapsItaly["maps_italy.py"]:::process
        MapNeighbour["map_neighbourhoods.py"]:::process
    end

    %% Presentation Layer
    subgraph "Presentation Layer"
        direction TB
        StreamlitApp["playground.py\n(Streamlit App)"]:::presentation
        PriceTrends["pages/PricesTrends.py"]:::presentation
        Config[".streamlit/config.toml"]:::presentation
        CSS["main.css"]:::presentation
        Icons["icons/"]:::presentation
    end

    %% End User
    subgraph "End User"
        direction TB
        Browser["User Browser"]:::external
    end

    %% Data Flows
    Immobi -->|"raw HTML"| Prepare
    Prepare -->|"upload raw/clean CSV"| GCSBucket
    Prepare -->|"write Parquet"| SalesParquet
    GCSBucket -->|"download Parquet"| SalesParquet
    SalesParquet -->|"load Parquet"| StreamlitApp

    ComuniExcel -->|"load Excel"| PriceMap
    CoordsCSV -->|"load CSV"| PriceMap
    ProvGeoJSON -->|"load GeoJSONs"| MapsItaly
    ItaliaMunic -->|"load Excel"| MapNeighbour
    MunCentroids -->|"load CSV"| MapNeighbour
    ProvItaliane -->|"load Excel"| MapNeighbour
    RegCentroids -->|"load CSV"| MapNeighbour

    Afford -->|"compute metrics"| SalesParquet
    PriceMap -->|"produce GeoDataFrame"| StreamlitApp
    MapsItaly -->|"produce GeoDataFrame"| StreamlitApp
    MapNeighbour -->|"produce GeoDataFrame"| StreamlitApp

    StreamlitApp -->|"renders charts & maps"| Browser
    PriceTrends -->|"module for trends page"| StreamlitApp

    %% Click Events
    click Prepare "https://github.com/tommella90/italy-house-prices/blob/main/prepare_data.py"
    click SalesParquet "https://github.com/tommella90/italy-house-prices/blob/main/dataframes/sales.parquet"
    click ComuniExcel "https://github.com/tommella90/italy-house-prices/blob/main/geodata/comuni.xlsx"
    click CoordsCSV "https://github.com/tommella90/italy-house-prices/blob/main/geodata/coords.csv"
    click ProvGeoJSON "https://github.com/tommella90/italy-house-prices/tree/main/geodata/prov_coords"
    click ItaliaMunic "https://github.com/tommella90/italy-house-prices/blob/main/geodata/italy_munic.xlsx"
    click MunCentroids "https://github.com/tommella90/italy-house-prices/blob/main/geodata/municipalities_centroids.csv"
    click ProvItaliane "https://github.com/tommella90/italy-house-prices/blob/main/geodata/province-italiane.xls"
    click RegCentroids "https://github.com/tommella90/italy-house-prices/blob/main/geodata/regions_centroids.csv"
    click Afford "https://github.com/tommella90/italy-house-prices/blob/main/affordability.py"
    click PriceMap "https://github.com/tommella90/italy-house-prices/blob/main/PricesMap.py"
    click MapsItaly "https://github.com/tommella90/italy-house-prices/blob/main/maps_italy.py"
    click MapNeighbour "https://github.com/tommella90/italy-house-prices/blob/main/map_neighbourhoods.py"
    click StreamlitApp "https://github.com/tommella90/italy-house-prices/blob/main/playground.py"
    click PriceTrends "https://github.com/tommella90/italy-house-prices/blob/main/pages/PricesTrends.py"
    click Config "https://github.com/tommella90/italy-house-prices/blob/main/.streamlit/config.toml"
    click CSS "https://github.com/tommella90/italy-house-prices/blob/main/main.css"
    click Icons "https://github.com/tommella90/italy-house-prices/tree/main/icons/"
    click README.md "https://github.com/tommella90/italy-house-prices/blob/main/README.md"
    click requirements.txt "https://github.com/tommella90/italy-house-prices/blob/main/requirements.txt"

    %% Styles
    classDef storage fill:#bbdefb,stroke:#1e88e5,color:#0d47a1
    classDef process fill:#c8e6c9,stroke:#43a047,color:#1b5e20
    classDef presentation fill:#ffcc80,stroke:#fb8c00,color:#e65100
    classDef external fill:#e0e0e0,stroke:#9e9e9e,color:#424242
```
