How lukasmfr/bookstoscrape_etl works
Main Components
• Scraping Module:
• Represented by scraping.py.
• This module extracts data from the Books to Scrape website using concurrent requests.
• Outputs raw data in CSV, JSON, and Excel formats (in the output folder) and also inserts data into a MySQL database.
• Database Component:
• The MySQL database (set up via database.sql) where the raw extracted data is stored.
• Data Cleaning & Analysis Module:
• Represented by data_cleaning_analysis.py.
• This component reads the scraped CSV data from the output folder, performs cleaning (type conversions, missing value handling, text normalization) and analysis, then writes the cleaned data back into a file (books_clean.csv in output).
• Data Visualization Module:
• Represented by data_visualization.py.
• Consumes the cleaned data to generate a variety of visualizations (bar charts, histograms, boxplots, violin plots, etc.).
• Outputs high-resolution images (stored in the images folder).
• Main Interactive Script:
• Represented by main.py.
• Acts as the coordinator that ties all components together via a menu-driven interface (leveraging the rich library for UI enhancements).
Relationships and Flow
• Start at the Scraping Module: Data is extracted from the external source (Books to Scrape website) and then pushed into two streams: one to generate flat file outputs (CSV, JSON, Excel) and the other to insert data into the MySQL database.
• Next, the Data Cleaning & Analysis Module pulls the raw data (primarily from the scraped CSV).
• The cleaned data, now containing transformed and normalized information, is saved as a separate file that is used by the Data Visualization Module.
• The Data Visualization Module then reads the cleaned data and generates insightful, high-resolution visual images that are stored in the images folder.
• The Main Interactive Script serves as an orchestrator, providing a user interface to run any or all of these steps as needed.
Architectural Patterns and Design Principles
• Modular Design: Each step of the ETL process is encapsulated in its own script.
• Layered Pipeline: You have a clear separation between data extraction, transformation (with cleaning), and visualization.
• Concurrency: In the scraping process, concurrent requests are used to improve performance.
• Data Persistence: The processed data is stored in multiple formats as well as in a dedicated MySQL database.
• Reusability and Extensibility: With distinct scripts for each stage, maintenance and further development (e.g., adding more types of visualizations or additional data transformations) are streamlined.
Technologies and Frameworks
• Python libraries: Requests, BeautifulSoup, Pandas, Matplotlib, Seaborn, MySQL Connector/Python, and Rich.
• MySQL database for persistent storage.
• Concurrency libraries such as concurrent.futures for parallel processing in scraping.