How mohamedaminebentayeb/scrape_visualize_data works
Project Type and Purpose
• A Python-based data-scraping and visualization tool.
• It contains multiple ETL pipelines (stock market, earthquakes, two weather sources) that fetch JSON data on a schedule, process/summarize it, and then provide a visualization layer via app.py.
Main Components to Model
A. Scheduler Layer
– Cron jobs (crontab.txt at root + bourse/crontab.txt) that trigger download scripts on a timetable.
B. ETL Pipelines (four domains)
1. Bourse (stock)
• Downloader: script_appel_api.sh → External Stock API
• Data folder: bourse/data/*.json
• Loader: script_mongo_db.py → MongoDB
2. Earthquakes
• Downloader: download_data.sh → USGS Earthquake API
• Storage: earthquakes/downloaded_data/*.json
• Summarizer: summarize_data.py
3. Meteo_Script (private meteoData)
• main.py & script.sh → Meteomatics API → meteo_script/meteoData/*.json
4. Open_Weather_Map
• Downloader: download_data.sh → OpenWeatherMap API
• Storage: open_weather_map/downloaded_data/*.json
• Summarizer: summarize_data.py
C. Data Storage
– Local File System (JSON directories per domain)
– MongoDB (used by the Bourse pipeline)
D. Visualization Service
– app.py (likely a Flask/FastAPI server)
– Reads from MongoDB and/or JSON files to render dashboards
– Serves to end-users’ browsers
E. Users
– Browser or any HTTP client consuming the visualization API/UI
Relationships & Data Flows
1. Cron Scheduler → triggers each domain’s download shell script
2. Shell Script → External API → pulls JSON → writes to local FS
3. Summarizer Script → reads raw JSON → produces summary outputs (either JSON files or direct DB writes)
4. Bourse Loader → writes cleaned data into MongoDB
5. app.py → queries MongoDB and/or reads local JSON → serves HTTP responses to the browser
Architectural Patterns & Design Principles
• ETL Pattern: clear Extraction, Transformation, (and Load for Bourse) stages
• Modular/Layered: separate pipelines by domain; decoupled through FS or DB
• Scheduler-Orchestrated: Cron handles timing, keeping pipelines independent
• Single Responsibility: each script does one thing (download, summarize, load, serve)