How portfolio-alvaro/proyecto-pulso-social-urbano works
Project type and purpose
• This is a data-processing + visualization prototype (full-stack Python service) that scrapes social media posts, performs sentiment and topic analysis, geolocates events, calculates KPIs over time windows, and exposes an interactive map & timeline dashboard via a Flask API. It’s orchestrated locally with Docker Compose.
File structure analysis
• Top level:
• urbano.py / version extendida con analisis espacio temporal.py – core scripts for ingestion & processing
• API Flask Dockerfile.py – API server entrypoint or Dockerfile (naming suggests combined file)
• Docker-compose.yml (and typo “Docker compose.yml”) – multi-container orchestration
• requirements.txt – Python dependencies (snscrape, VADER/TextBlob, Folium, TimestampedGeoJson, Postgres driver, etc.)
• README.md – roadmap & feature description
• Estructura de proyecto – likely contains deeper folder structure (not shown)
• Indicators: single-repo, phase-1 MVP, monolithic Flask + processing pipeline, Postgres persistence, Folium/Leaflet frontend.
Key insights from README
• Phase-1 MVP pipeline:
• Ingest: snscrape → filter city/hashtags
• Process: sentiment analysis (VADER/TextBlob), NMF topic clustering on TF-IDF, simple geolocation (geopy)
• Store: Postgres (with basic geo-indexes)
• Expose: Flask API, interactive maps (HeatMapWithTime & TimestampedGeoJson)
• Docker Compose orchestrates (Scraper, Processor/API, Postgres, maybe UI server).
Main components & interactions
a. Data Ingestion Service
• Inputs: external social-media (snscrape)
• Output: raw posts
b. Processing Pipeline
• Sentiment Analyzer (VADER/TextBlob)
• Topic Clustering (NMF w/ TF-IDF)
• Geolocator (geopy → district aggregates)
• KPI Calculator (velocity, coverage over sliding window)
• Outputs processed records & geojson payloads
c. Persistence Layer
• Postgres DB (tables: raw_posts, analyzed_posts, topics, geodata, kpis)
d. API Layer
• Flask app serving endpoints: /heatmap, /timeline, /kpis
• Reads from Postgres, returns JSON/geojson
e. Dashboard / Frontend
• Folium/Leaflet templates served via Flask or static files
• Components: HeatMapWithTime, TimestampedGeoJson timeline
f. Orchestration
• Docker Compose: defines containers for each service + Postgres
Architectural patterns & principles
• Pipeline architecture (linear ETL: ingest → transform → load → serve)
• Separation of concerns: ingestion, processing, storage, API, visualization
• Containerization for reproducibility (Docker Compose)
• Future roadmap: move to microservices for each stage, introduce feature store & model registry