How tommella90/italy-house-prices works
Project type and purpose
• This is a Python-based ETL + interactive data visualization app.
• It scrapes public housing listings from Immobiliare.it, stores and processes them (ETL), then exposes an interactive Streamlit front-end for exploring Italy house prices by city, region, and over time.
Main components
a. External Data Source
• Immobiliare.it (HTML pages of listings)
b. Scraping & Ingestion (ETL)
• prepare_data.py – orchestrates HTTP requests, parsing, initial cleaning
• (Implicit scraping logic) → raw CSV/JSON or DataFrame
c. Cloud Storage
• Google Cloud Storage “italy-housing-data” (raw and processed objects)
d. Data Warehouse / Local Data Store
• dataframes/sales.parquet – unified sales dataset
• geodata/… – Excel/CSV/GeoJSON for municipalities, provinces, centroids
e. Data Processing & Enrichment
• affordability.py – computes affordability metrics
• PricesMap.py, maps_italy.py, map_neighbourhoods.py – prepare GeoDataFrames for mapping
f. Interactive Web App (Presentation Layer)
• Streamlit app entrypoint (e.g., playground.py or main script)
• pages/PricesTrends.py – multipage navigation
g. End User
• Browser/UI interacting with Streamlit
Relationships & Data Flows
1. Immobiliare.it → prepare_data.py (scrapes raw housing data)
2. prepare_data.py → GCS bucket (uploads raw/clean CSV or Parquet)
3. ETL scripts (prepare_data.py + affordability.py) → dataframes/sales.parquet
4. Geo-assets (geodata/) loaded locally by mapping scripts
5. Streamlit App loads:
• sales.parquet (Pandas/Parquet)
• GeoJSON/CSV/Excel from geodata/
6. Streamlit UI renders maps & charts → User
Architectural Patterns & Design Principles
• ETL pipeline → Data Lake → Self-service BI (Streamlit)
• Separation of concerns:
• Ingestion & storage
• Processing & enrichment
• Presentation
Must-Include
• Clear labels on every component (script name, file store, service)