# lukasmfr/bookstoscrape_etl architecture

> Architecture diagram of the GitHub repository lukasmfr/bookstoscrape_etl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lukasmfr/bookstoscrape_etl
- Repository: https://github.com/lukasmfr/bookstoscrape_etl

## Overview

##### Main Components
- Scraping Module:
- Represented by scraping.py.
- This module extracts data from the Books to Scrape website using concurrent requests.
- Outputs raw data in CSV, JSON, and Excel formats (in the output folder) and also inserts data into a MySQL database.
- Database Component:
- The MySQL database (set up via database.sql) where the raw extracted data is stored.
- Data Cleaning & Analysis Module:
- Represented by data_cleaning_analysis.py.
- This component reads the scraped CSV data from the output folder, performs cleaning (type conversions, missing value handling, text normalization) and analysis, then writes the cleaned data back into a file (books_clean.csv in output).
- Data Visualization Module:
- Represented by data_visualization.py.
- Consumes the cleaned data to generate a variety of visualizations (bar charts, histograms, boxplots, violin plots, etc.).
- Outputs high-resolution images (stored in the images folder).
- Main Interactive Script:
- Represented by main.py.
- Acts as the coordinator that ties all components together via a menu-driven interface (leveraging the rich library for UI enhancements).

##### Relationships and Flow
- Start at the Scraping Module: Data is extracted from the external source (Books to Scrape website) and then pushed into two streams: one to generate flat file outputs (CSV, JSON, Excel) and the other to insert data into the MySQL database.
- Next, the Data Cleaning & Analysis Module pulls the raw data (primarily from the scraped CSV).
- The cleaned data, now containing transformed and normalized information, is saved as a separate file that is used by the Data Visualization Module.
- The Data Visualization Module then reads the cleaned data and generates insightful, high-resolution visual images that are stored in the images folder.
- The Main Interactive Script serves as an orchestrator, providing a user interface to run any or all of these steps as needed.

##### Architectural Patterns and Design Principles
- Modular Design: Each step of the ETL process is encapsulated in its own script.
- Layered Pipeline: You have a clear separation between data extraction, transformation (with cleaning), and visualization.
- Concurrency: In the scraping process, concurrent requests are used to improve performance.
- Data Persistence: The processed data is stored in multiple formats as well as in a dedicated MySQL database.
- Reusability and Extensibility: With distinct scripts for each stage, maintenance and further development (e.g., adding more types of visualizations or additional data transformations) are streamlined.

##### Technologies and Frameworks
- Python libraries: Requests, BeautifulSoup, Pandas, Matplotlib, Seaborn, MySQL Connector/Python, and Rich.
- MySQL database for persistent storage.
- Concurrency libraries such as concurrent.futures for parallel processing in scraping.

## Diagram (Mermaid)

```mermaid
flowchart TD
    A["Main Interactive Script (main.py)"]:::script
    B["Scraping Module (scraping.py)"]:::script
    D["Data Cleaning & Analysis (data_cleaning_analysis.py)"]:::script
    G["Data Visualization (data_visualization.py)"]:::script
    F["Database"]:::db
    cd["Cleaned Data (books_clean.csv)"]:::file
    img["Images Folder"]:::file

    A -->|"orchestrate"| B
    A -->|"orchestrate"| D
    A -->|"orchestrate"| G

    B -->|"flat_files"| csv
    B -->|"insert_DB"| F

    subgraph "Raw Output Files"
        csv["CSV (books.csv)"]:::file
        json["JSON (books.json)"]:::file
        excel["Excel (books.xlsx)"]:::file
    end

    csv -->|"clean_data"| D
    D -->|"create_cleaned_file"| cd
    cd -->|"generate_visualizations"| G
    G -->|"save_images"| img

    %% Click Events
    click A "https://github.com/lukasmfr/bookstoscrape_etl/blob/main/main.py"
    click B "https://github.com/lukasmfr/bookstoscrape_etl/blob/main/scraping.py"
    click csv "https://github.com/lukasmfr/bookstoscrape_etl/blob/main/output/books.csv"
    click json "https://github.com/lukasmfr/bookstoscrape_etl/blob/main/output/books.json"
    click excel "https://github.com/lukasmfr/bookstoscrape_etl/blob/main/output/books.xlsx"
    click F "https://github.com/lukasmfr/bookstoscrape_etl/blob/main/database.sql"
    click D "https://github.com/lukasmfr/bookstoscrape_etl/blob/main/data_cleaning_analysis.py"
    click cd "https://github.com/lukasmfr/bookstoscrape_etl/blob/main/output/books_clean.csv"
    click G "https://github.com/lukasmfr/bookstoscrape_etl/blob/main/data_visualization.py"
    click img "https://github.com/lukasmfr/bookstoscrape_etl/tree/main/images"

    %% Styles
    classDef script fill:#AEDFF7,stroke:#000,stroke-width:2px;
    classDef file fill:#E2F0CE,stroke:#000,stroke-width:2px;
    classDef db fill:#F7D6AE,stroke:#000,stroke-width:2px;
```
