How guilhermecf10/uerj-mat-discreta-2 works
Project Type and Purpose
• This is a CLI-driven data-analysis/ETL tool (not a full-stack web app).
• Purpose: scrape bus-stop data from Moovit, geocode stops, build a network graph, and render an interactive map.
Main Components
a. AppController (script/main.py)
– Orchestrates the workflow and handles CLI flags (force_rescrape, force_regeocode).
b. MoovitScraper (script/moovit_scraper.py)
– Fetches raw stop data from Moovit pages.
c. GeoCoder (script/geocoder.py)
– Calls Google Maps Geocoding API; enriches stops with lat/lon.
d. DataExporter (script/data_exporter.py)
– Reads/writes CSVs (raw vs. geocoded).
e. GraphBuilder & Analysis (script/graph_analysis.py)
– Constructs NetworkX DiGraph, computes distances, caching.
f. MapGenerator (part of graph_analysis.py)
– Produces interactive HTML map via Folium.
g. Cache & Data Stores
– /script/cache/cached_moovit_graph.gpickle
– /script/data/*.csv
h. External Services
– Moovit website (HTML pages)
– Google Maps Geocoding API
i. Tests (script/tests/…)
– Example inputs + expected outputs to validate scrapers and mapping.
Relationships & Data Flows
1. AppController invokes MoovitScraper → outputs moovit_stops_raw.csv → stored in script/data.
2. If force_regeocode or missing geocoded CSV, AppController calls GeoCoder → reads raw CSV → writes moovit_stops_geocoded.csv.
3. DataExporter abstracts CSV I/O between modules.
4. GraphBuilder reads geocoded CSV → checks for cached GPickle → if absent/forced, builds NetworkX DiGraph (nodes=stops, edges=connections) → writes cache.
5. MapGenerator reads graph → draws markers & edges → outputs map_moovit_stops.html.
6. AppController exits, returning path to map.
7. Tests invoke subsets of these modules with controlled data and validate HTML or map files.
Architectural Patterns & Principles
• Modular separation: each concern in its own module/class (Single Responsibility Principle).
• Pipeline/ETL flow: linear stages with caching at key points.
• Use of caching for idempotency and performance.
• CLI orchestration (flags to toggle pipelines).
• External-service abstraction (API key via ENV).