# zakariaeyahya/amazon architecture

> Architecture diagram of the GitHub repository zakariaeyahya/amazon, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zakariaeyahya/amazon
- Repository: https://github.com/zakariaeyahya/amazon

## Overview

##### Project Type & High-Level Purpose
- Type: Distributed full-stack data-pipeline for web scraping, ETL and analytics.
- Purpose: Crawl Amazon product pages (metadata, taxonomy, reviews), process & clean data, load into Snowflake for analytics, and expose insights via Jupyter notebooks, word clouds and a Streamlit-based visualization.

##### Main Architectural Components
a. Orchestration Layer
- Apache Airflow (docker/airflow): scheduler + workers executing DAGs (airflow_dags/amazon_scraping_dag.py).
b. Extraction Layer
- Python modules under extraction/: category discovery, product details scraper, reviews scraper, comment parser, plus upload_s3.py for optional AWS S3 staging.
- Anti-blocking utilities (utils/request_utils.py, scraping_logger.py, scraping_metrics.py).
c. Storage & Staging
- Local file system / CSV/JSON (data/, transformations/data_clean/, transformations/datafinal.csv).
- Optional S3 bucket for inter-task staging (via upload_s3.py).
d. Transformation & Loading
- Python ETL scripts in transformations/ (extract_data.py, snowflakeconnection.py, SQLrequet.sql).
- Snowflake data warehouse (tables, views, schema under transformations/snowflake/).
e. Analysis & Visualization
- Sentiment-analysis Jupyter notebook (Analyse_des_sentiments/analyse_sentiments.ipynb) and generate_wordcloud.py.
- Streamlit app in visualisation/ producing dashboards.
f. Infrastructure
- Docker Compose (docker-compose.yml) orchestrates Airflow, Selenium (docker/selenium) and other services.
- Configuration in config/scraping_config.yaml and environment variables for Snowflake credentials.
g. Observability
- Logging & metrics collected by utils/logger + utils/metrics and exposed alongside Airflow logs.

##### Relationships & Data Flow
1. External Trigger → Airflow DAG scheduler
2. DAG tasks invoke extraction modules (extraction/*.py), fetch from Amazon (external service)
3. Scraped raw data written to local FS or S3
4. Transformation scripts read raw data → clean/normalize → write cleaned CSV
5. Snowflake connector loads cleaned data into Snowflake tables/views
6. Data scientists run Jupyter notebooks and Streamlit app against Snowflake to generate reports, word clouds, dashboards

##### Architectural Patterns & Design Principles
- Pipeline orchestration (Airflow DAGs).
- Modular extraction libraries (single-responsibility modules under extraction/).
- Staging area for raw vs. cleaned data (separation of raw storage vs. ETL output).
- Configuration-driven pipelines (YAML + .env for credentials).
- Containerization for reproducibility (Docker + Docker-Compose).
- ThreadPoolExecutor for parallel scraping.

##### Essential Details
- Docker-Compose relationships: link “Airflow” container to “Selenium” container for headless browser tasks.
- Call out config service (config/scraping_config.yaml) feeding both Airflow and extraction modules.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Presentation/Visualization Layer
    subgraph "Presentation/Visualization"
        direction TB
        JN["Jupyter Notebook"]:::compute
        ST["Streamlit App"]:::compute
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        direction TB
        AS["Airflow Scheduler"]:::compute
        AW["Airflow Worker"]:::compute
    end

    %% Extraction Layer
    subgraph "Extraction Layer"
        direction TB
        CD["Category Discovery"]:::compute
        PD["Product Details Scraper"]:::compute
        RS["Reviews Scraper"]:::compute
        CP["Comment Parser"]:::compute
        US3["Upload to S3"]:::compute
        RU["Request Utils"]:::compute
        SL["Scraping Logger"]:::compute
        SM["Scraping Metrics"]:::compute
    end

    %% Storage & Staging Layer
    subgraph "Storage & Staging"
        direction TB
        LF["Local File System"]:::store
        S3["S3 Bucket (Optional)"]:::store
    end

    %% Transformation & Loading Layer
    subgraph "Transformation & Loading"
        direction TB
        ED["extract_data.py"]:::compute
        SFConn["Snowflake Connector"]:::compute
        SQLQ["SQLRequet.sql"]:::compute
        SF["Snowflake Warehouse"]:::store
    end

    %% External Services
    subgraph "External Services"
        direction TB
        AMZ["Amazon Site"]:::external
        PROXY["Proxy/User-Agent Rotation"]:::external
    end

    %% Configuration
    CFG["scraping_config.yaml"]:::external

    %% Observability
    OBSLOG["Logging & Metrics"]:::compute

    %% Connections
    %% Trigger
    trigger(("External Trigger")):::external -->|triggers DAG| AS
    AS -->|schedules DAG| AW

    %% DAG to Extraction
    AW -->|calls amazon_scraping_dag.py| CD
    AW --> PD
    AW --> RS
    AW --> CP

    %% Utils used by scrapers
    CD --> RU
    PD --> RU
    RS --> RU
    CP --> RU
    CD --> SL
    PD --> SL
    RS --> SL
    CP --> SL
    CD --> SM
    PD --> SM
    RS --> SM
    CP --> SM

    %% Extraction to Storage
    CD -->|raw JSON/CSV| LF
    PD -->|raw CSV| LF
    RS -->|raw CSV| LF
    CP -->|parsed comments| LF
    US3 -->|S3 PUT| S3
    LF -->|optionally upload| US3

    %% Storage to Transformation
    LF -->|reads raw data| ED
    S3 -->|reads raw data| ED

    %% Transformation to Data Warehouse
    ED -->|cleaned CSV| SFConn
    SFConn -->|SQL load| SF

    %% Data Warehouse to Visualization
    SF -->|queries via Python/SQL| JN
    SF -->|queries via Python| ST

    %% Config injection
    CFG --> AS
    CFG --> CD
    CFG --> PD
    CFG --> RS
    CFG --> CP

    %% Observability connections
    SL --> OBSLOG
    SM --> OBSLOG
    AS --> OBSLOG
    AW --> OBSLOG

    %% Containers
    subgraph "Docker Compose"
        direction TB
        DC["docker-compose.yml"]:::external
        AIRCONT["Airflow Container"]:::compute
        SELCONT["Selenium Container"]:::compute
    end
    DC --> AIRCONT
    DC --> SELCONT
    AIRCONT --> AS
    AIRCONT --> AW
    SELCONT --> PD

    %% External service connection
    PD -->|HTTP requests| AMZ
    RS -->|HTTP requests| AMZ
    CP -->|HTTP requests| AMZ
    PD --> PROXY
    RS --> PROXY
    CP --> PROXY

    %% Click Events
    click CD "https://github.com/zakariaeyahya/amazon/blob/main/extraction/categories.py"
    click PD "https://github.com/zakariaeyahya/amazon/blob/main/extraction/amazon_details_scraper.py"
    click RS "https://github.com/zakariaeyahya/amazon/blob/main/extraction/scraping.py"
    click CP "https://github.com/zakariaeyahya/amazon/blob/main/extraction/commentaire.py"
    click US3 "https://github.com/zakariaeyahya/amazon/blob/main/extraction/upload_s3.py"
    click RU "https://github.com/zakariaeyahya/amazon/blob/main/utils/request_utils.py"
    click SL "https://github.com/zakariaeyahya/amazon/blob/main/utils/scraping_logger.py"
    click SM "https://github.com/zakariaeyahya/amazon/blob/main/utils/scraping_metrics.py"
    click AS "https://github.com/zakariaeyahya/amazon/tree/main/docker/airflow"
    click AS "https://github.com/zakariaeyahya/amazon/tree/main/docker/airflow/Dockerfile"
    click AW "https://github.com/zakariaeyahya/amazon/blob/main/airflow_dags/amazon_scraping_dag.py"
    click DC "https://github.com/zakariaeyahya/amazon/blob/main/docker-compose.yml"
    click ED "https://github.com/zakariaeyahya/amazon/blob/main/transformations/extract_data.py"
    click SFConn "https://github.com/zakariaeyahya/amazon/blob/main/transformations/snowflake/snowflakeconnection.py"
    click SQLQ "https://github.com/zakariaeyahya/amazon/blob/main/transformations/snowflake/SQLrequet.sql"
    click JN "https://github.com/zakariaeyahya/amazon/blob/main/Analyse_des_sentiments/analyse_sentiments.ipynb"
    click genWC "https://github.com/zakariaeyahya/amazon/blob/main/Analyse_des_sentiments/generate_wordcloud.py"
    click ST "https://github.com/zakariaeyahya/amazon/blob/main/visualisation/viz.py"
    click streamGen "https://github.com/zakariaeyahya/amazon/blob/main/visualisation/generate_wordcloud.py"
    click streamConfig "https://github.com/zakariaeyahya/amazon/blob/main/visualisation/.streamlit/config.toml"
    click CFG "https://github.com/zakariaeyahya/amazon/blob/main/config/scraping_config.yaml"

    %% Styles
    classDef compute fill:#E0F7FA,stroke:#006064
    classDef store fill:#E8F5E9,stroke:#2E7D32
    classDef external fill:#ECEFF1,stroke:#607D8B
```
