How zakariaeyahya/amazon works
Project Type & High-Level Purpose
• Type: Distributed full-stack data-pipeline for web scraping, ETL and analytics.
• Purpose: Crawl Amazon product pages (metadata, taxonomy, reviews), process & clean data, load into Snowflake for analytics, and expose insights via Jupyter notebooks, word clouds and a Streamlit-based visualization.
Main Architectural Components
a. Orchestration Layer
• Apache Airflow (docker/airflow): scheduler + workers executing DAGs (airflow_dags/amazon_scraping_dag.py).
b. Extraction Layer
• Python modules under extraction/: category discovery, product details scraper, reviews scraper, comment parser, plus upload_s3.py for optional AWS S3 staging.
• Anti-blocking utilities (utils/request_utils.py, scraping_logger.py, scraping_metrics.py).
c. Storage & Staging
• Local file system / CSV/JSON (data/, transformations/data_clean/, transformations/datafinal.csv).
• Optional S3 bucket for inter-task staging (via upload_s3.py).
d. Transformation & Loading
• Python ETL scripts in transformations/ (extract_data.py, snowflakeconnection.py, SQLrequet.sql).
• Snowflake data warehouse (tables, views, schema under transformations/snowflake/).
e. Analysis & Visualization
• Sentiment-analysis Jupyter notebook (Analyse_des_sentiments/analyse_sentiments.ipynb) and generate_wordcloud.py.
• Streamlit app in visualisation/ producing dashboards.
f. Infrastructure
• Docker Compose (docker-compose.yml) orchestrates Airflow, Selenium (docker/selenium) and other services.
• Configuration in config/scraping_config.yaml and environment variables for Snowflake credentials.
g. Observability
• Logging & metrics collected by utils/logger + utils/metrics and exposed alongside Airflow logs.
Relationships & Data Flow
1. External Trigger → Airflow DAG scheduler
2. DAG tasks invoke extraction modules (extraction/*.py), fetch from Amazon (external service)
3. Scraped raw data written to local FS or S3
4. Transformation scripts read raw data → clean/normalize → write cleaned CSV
5. Snowflake connector loads cleaned data into Snowflake tables/views
6. Data scientists run Jupyter notebooks and Streamlit app against Snowflake to generate reports, word clouds, dashboards
Architectural Patterns & Design Principles
• Pipeline orchestration (Airflow DAGs).
• Modular extraction libraries (single-responsibility modules under extraction/).
• Staging area for raw vs. cleaned data (separation of raw storage vs. ETL output).
• Configuration-driven pipelines (YAML + .env for credentials).
• Containerization for reproducibility (Docker + Docker-Compose).
• ThreadPoolExecutor for parallel scraping.
Essential Details
• Docker-Compose relationships: link “Airflow” container to “Selenium” container for headless browser tasks.
• Call out config service (config/scraping_config.yaml) feeding both Airflow and extraction modules.