# mohamedaminebentayeb/scrape_visualize_data architecture

> Architecture diagram of the GitHub repository mohamedaminebentayeb/scrape_visualize_data, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mohamedaminebentayeb/scrape_visualize_data
- Repository: https://github.com/mohamedaminebentayeb/scrape_visualize_data

## Overview

##### Project Type and Purpose
- A Python-based data-scraping and visualization tool.
- It contains multiple ETL pipelines (stock market, earthquakes, two weather sources) that fetch JSON data on a schedule, process/summarize it, and then provide a visualization layer via app.py.

##### Main Components to Model
A. Scheduler Layer
– Cron jobs (crontab.txt at root + bourse/crontab.txt) that trigger download scripts on a timetable.
B. ETL Pipelines (four domains)
1. Bourse (stock)
- Downloader: script_appel_api.sh → External Stock API
- Data folder: bourse/data/*.json
- Loader: script_mongo_db.py → MongoDB
2. Earthquakes
- Downloader: download_data.sh → USGS Earthquake API
- Storage: earthquakes/downloaded_data/*.json
- Summarizer: summarize_data.py
3. Meteo_Script (private meteoData)
- main.py & script.sh → Meteomatics API → meteo_script/meteoData/*.json
4. Open_Weather_Map
- Downloader: download_data.sh → OpenWeatherMap API
- Storage: open_weather_map/downloaded_data/*.json
- Summarizer: summarize_data.py
C. Data Storage
– Local File System (JSON directories per domain)
– MongoDB (used by the Bourse pipeline)
D. Visualization Service
– app.py (likely a Flask/FastAPI server)
– Reads from MongoDB and/or JSON files to render dashboards
– Serves to end-users’ browsers
E. Users
– Browser or any HTTP client consuming the visualization API/UI

##### Relationships & Data Flows
1. Cron Scheduler → triggers each domain’s download shell script
2. Shell Script → External API → pulls JSON → writes to local FS
3. Summarizer Script → reads raw JSON → produces summary outputs (either JSON files or direct DB writes)
4. Bourse Loader → writes cleaned data into MongoDB
5. app.py → queries MongoDB and/or reads local JSON → serves HTTP responses to the browser

##### Architectural Patterns & Design Principles
- ETL Pattern: clear Extraction, Transformation, (and Load for Bourse) stages
- Modular/Layered: separate pipelines by domain; decoupled through FS or DB
- Scheduler-Orchestrated: Cron handles timing, keeping pipelines independent
- Single Responsibility: each script does one thing (download, summarize, load, serve)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Scheduler Layer
    subgraph "Scheduler Layer"
        CRON_ROOT("crontab.txt"):::scheduler
        CRON_BOURSE("bourse/crontab.txt"):::scheduler
    end

    %% Bourse Pipeline
    subgraph "Bourse Pipeline"
        B_SCRIPT["script_appel_api.sh"]:::script
        B_DATA["bourse/data/"]:::storage
        B_LOADER["script_mongo_db.py"]:::script
        MONGO[(MongoDB)]:::db
        EX_STOCK["External Stock API"]:::external
    end

    %% Earthquake Pipeline
    subgraph "Earthquake Pipeline"
        E_SCRIPT["earthquakes/download_data.sh"]:::script
        E_DATA["earthquakes/downloaded_data/"]:::storage
        E_SUM["earthquakes/summarize_data.py"]:::script
        EX_USGS["USGS Earthquake API"]:::external
    end

    %% Meteo_Script Pipeline
    subgraph "Meteo_Script Pipeline"
        MS_SCRIPT["meteo_script/script.sh"]:::script
        MS_MAIN["meteo_script/main.py"]:::script
        MS_DATA["meteo_script/meteoData/"]:::storage
        EX_METEO["Meteomatics API"]:::external
    end

    %% Open Weather Map Pipeline
    subgraph "Open_Weather_Map Pipeline"
        OW_SCRIPT["open_weather_map/download_data.sh"]:::script
        OW_DATA["open_weather_map/downloaded_data/"]:::storage
        OW_SUM["open_weather_map/summarize_data.py"]:::script
        EX_OWM["OpenWeatherMap API"]:::external
    end

    %% Visualization Service
    subgraph "Visualization Service"
        APP["app.py"]:::web
        USER["User Browser"]:::user
    end

    %% Scheduler triggers
    CRON_ROOT -->|cron trigger| E_SCRIPT
    CRON_ROOT -->|cron trigger| MS_SCRIPT
    CRON_ROOT -->|cron trigger| OW_SCRIPT
    CRON_BOURSE -->|cron trigger| B_SCRIPT

    %% Bourse flow
    B_SCRIPT -->|"HTTP GET JSON"| EX_STOCK
    B_SCRIPT -->|"write JSON"| B_DATA
    B_LOADER -->|"read JSON"| B_DATA
    B_LOADER -->|"DB insert"| MONGO

    %% Earthquake flow
    E_SCRIPT -->|"HTTP GET JSON"| EX_USGS
    E_SCRIPT -->|"write JSON"| E_DATA
    E_SUM -->|"read JSON"| E_DATA
    E_SUM -->|"write summary JSON"| E_DATA

    %% Meteo flow
    MS_SCRIPT -->|"HTTP GET JSON"| EX_METEO
    MS_SCRIPT -->|"write JSON"| MS_DATA
    MS_MAIN -->|"process JSON"| MS_DATA

    %% Open Weather flow
    OW_SCRIPT -->|"HTTP GET JSON"| EX_OWM
    OW_SCRIPT -->|"write JSON"| OW_DATA
    OW_SUM -->|"read JSON"| OW_DATA
    OW_SUM -->|"write summary JSON"| OW_DATA

    %% Visualization flow
    APP -->|"query"| MONGO
    APP -->|"read JSON"| B_DATA
    APP -->|"read JSON"| E_DATA
    APP -->|"read JSON"| MS_DATA
    APP -->|"read JSON"| OW_DATA
    APP -->|HTTP response| USER

    %% Click Events
    click CRON_ROOT "https://github.com/mohamedaminebentayeb/scrape_visualize_data/blob/main/crontab.txt"
    click CRON_BOURSE "https://github.com/mohamedaminebentayeb/scrape_visualize_data/blob/main/bourse/crontab.txt"
    click B_SCRIPT "https://github.com/mohamedaminebentayeb/scrape_visualize_data/blob/main/bourse/script_appel_api.sh"
    click B_DATA "https://github.com/mohamedaminebentayeb/scrape_visualize_data/tree/main/bourse/data/"
    click B_LOADER "https://github.com/mohamedaminebentayeb/scrape_visualize_data/blob/main/bourse/script_mongo_db.py"
    click E_SCRIPT "https://github.com/mohamedaminebentayeb/scrape_visualize_data/blob/main/earthquakes/download_data.sh"
    click E_DATA "https://github.com/mohamedaminebentayeb/scrape_visualize_data/tree/main/earthquakes/downloaded_data/"
    click E_SUM "https://github.com/mohamedaminebentayeb/scrape_visualize_data/blob/main/earthquakes/summarize_data.py"
    click MS_SCRIPT "https://github.com/mohamedaminebentayeb/scrape_visualize_data/blob/main/meteo_script/script.sh"
    click MS_MAIN "https://github.com/mohamedaminebentayeb/scrape_visualize_data/blob/main/meteo_script/main.py"
    click MS_DATA "https://github.com/mohamedaminebentayeb/scrape_visualize_data/tree/main/meteo_script/meteoData/"
    click OW_SCRIPT "https://github.com/mohamedaminebentayeb/scrape_visualize_data/blob/main/open_weather_map/download_data.sh"
    click OW_DATA "https://github.com/mohamedaminebentayeb/scrape_visualize_data/tree/main/open_weather_map/downloaded_data/"
    click OW_SUM "https://github.com/mohamedaminebentayeb/scrape_visualize_data/blob/main/open_weather_map/summarize_data.py"
    click APP "https://github.com/mohamedaminebentayeb/scrape_visualize_data/blob/main/app.py"

    %% Styles
    classDef scheduler fill:#f9e79f,stroke:#b7950b
    classDef script fill:#aed6f1,stroke:#2980b9
    classDef storage fill:#a9dfbf,stroke:#27ae60
    classDef db fill:#f5cba7,stroke:#d35400
    classDef web fill:#d7bde2,stroke:#8e44ad
    classDef user fill:#ebedef,stroke:#7b7d7d
    classDef external fill:#f2f3f4,stroke:#7f8c8d,stroke-dasharray: 5 5
```
