# rivaquiroga/datapalooza-2025 architecture

> Architecture diagram of the GitHub repository rivaquiroga/datapalooza-2025, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rivaquiroga/datapalooza-2025
- Repository: https://github.com/rivaquiroga/datapalooza-2025

## Overview

A. Project Overview and Type
- This repository is designed as a set of hands-on workshop exercises for “Web scraping con Python” from Datapalooza 2025.
- It contains several independent Python scripts, each targeting a specific web scraping task (e.g., extract text, extract tables, parse links and download files, and a Selenium demo for dynamic sites).
- It is not a full-stack application but more of an educational/demonstration resource for web scraping.

B. Analyzing the File Structure and Key Components
- The top-level files include a README and multiple Python scripts (ejercicio-1_texto.py, ejercicio-2_tablas.py, ejercicio-3_enlaces-descargas.py, ejercicio-4_demo-selenium.py), which represent separate exercises.
- The README file provides context, instructions for installing dependencies (requests, beautifulsoup4, pandas, lxml, and Selenium for dynamic websites) and links to example websites.
- The project uses common external libraries for web scraping (Beautiful Soup, Requests, and Selenium), which should be represented as critical dependencies.

##### Main Components
- Workshop Controller: Represent this as the “Instructor/Workshop Flow” that guides through the exercises.
- External Libraries/Dependencies: Include components/libraries such as:
- Requests & Beautiful Soup for handling HTTP requests and parsing HTML (for static sites)
- Selenium (with its WebDriver) for handling dynamic content in web pages.
- Pandas & lxml for data manipulation and parsing structured data like tables.

##### Relationships and Interactions
- Each Python script (e.g., ejercicio-1_texto.py) can be shown as a module that initiates an HTTP request to a target website.
- Data flow from Target Website to the scraping module using “HTTP Requests”.
- Flow from the module to the external libraries (requests, Beautiful Soup or Selenium) that process the HTML.
- Finally, output results (printed text, extracted tables, downloaded files).

##### Architectural Patterns and Design Principles
- Although not following patterns like MVC or microservices, note the separation of concerns: each script handles its specific task and leverages modular libraries.
- That the architecture is “script-based” where every exercise is an independent processing unit, demonstrating various scraping techniques.
- Reusability concepts by showing that the external libraries are common across modules.

- For each module, include sub-components:
- Entry (HTTP request initiation)
- Parsing/Extraction (processing with Beautiful Soup or Selenium WebDriver)
- Output (results display or file download)

- Component is clearly labeled:
- Use clear descriptive labels such as “HTTP Request Handler”, “HTML Parser – BeautifulSoup”, or “Dynamic Content Handler – Selenium”.
- Differentiate between external dependencies (e.g., library components) and internal modules (each exercise).
- Distinguish between static (e.g., beautifulsoup4-based) and dynamic (e.g., Selenium-based) scraping approaches.
- To capture essential elements: dependencies, external targets, and the processing steps for each exercise.

F. Final Steps for the Principal Software Engineer
- Review the requirements for the depicted dependencies from the README and include any specific installation notes or usage details if necessary.

## Diagram (Mermaid)

```mermaid
flowchart TD
    WC["Workshop Controller"]:::workshopNode

    subgraph "Exercise Modules"
        subgraph "Ejercicio 1: Texto"
            E1_Req["HTTP Request (Text)"]:::exercise
            E1_Parse["Parsing (BeautifulSoup)"]:::exercise
            E1_Out["Output (Text)"]:::exercise
        end
        subgraph "Ejercicio 2: Tablas"
            E2_Req["HTTP Request (Tables)"]:::exercise
            E2_Parse["Parsing (Pandas & lxml)"]:::exercise
            E2_Out["Output (Table Data)"]:::exercise
        end
        subgraph "Ejercicio 3: Enlaces y Descargas"
            E3_Req["HTTP Request (Links)"]:::exercise
            E3_Parse["Parsing (BeautifulSoup)"]:::exercise
            E3_Out["Output (Downloads)"]:::exercise
        end
        subgraph "Ejercicio 4: Demo Selenium"
            E4_Req["HTTP Request (Dynamic)"]:::exercise
            E4_Parse["Processing (Selenium WebDriver)"]:::exercise
            E4_Out["Output (Dynamic Content)"]:::exercise
        end
    end

    subgraph "External Systems and Dependencies"
        StaticWeb["Static Web Page"]:::extSystem
        DynamicWeb["Dynamic Web Page"]:::extSystem
        RequestsLib["Requests Library"]:::dependency
        BeautifulSoupNode["HTML Parser (BeautifulSoup)"]:::dependency
        PandasLxmlNode["Data Parser (Pandas & lxml)"]:::dependency
        SeleniumWDNode["Dynamic Content Handler (Selenium WebDriver)"]:::dependency
    end

    %% Connections from Workshop Controller to Exercise Modules
    WC --> E1_Req
    WC --> E2_Req
    WC --> E3_Req
    WC --> E4_Req

    %% Internal flow for each exercise module
    E1_Req --> E1_Parse
    E1_Parse --> E1_Out

    E2_Req --> E2_Parse
    E2_Parse --> E2_Out

    E3_Req --> E3_Parse
    E3_Parse --> E3_Out

    E4_Req --> E4_Parse
    E4_Parse --> E4_Out

    %% External websites feeding into HTTP Request steps
    StaticWeb --- E1_Req
    StaticWeb --- E2_Req
    StaticWeb --- E3_Req
    DynamicWeb --- E4_Req

    %% Dependencies for processing modules
    E1_Req --- RequestsLib
    E1_Parse --- BeautifulSoupNode

    E2_Req --- RequestsLib
    E2_Parse --- PandasLxmlNode

    E3_Req --- RequestsLib
    E3_Parse --- BeautifulSoupNode

    E4_Req --- SeleniumWDNode
    E4_Parse --- SeleniumWDNode

    %% Click Events
    click WC "https://github.com/rivaquiroga/datapalooza-2025/blob/main/README.md"
    click E1_Req "https://github.com/rivaquiroga/datapalooza-2025/blob/main/ejercicio-1_texto.py"
    click E2_Req "https://github.com/rivaquiroga/datapalooza-2025/blob/main/ejercicio-2_tablas.py"
    click E3_Req "https://github.com/rivaquiroga/datapalooza-2025/blob/main/ejercicio-3_enlaces-descargas.py"
    click E4_Req "https://github.com/rivaquiroga/datapalooza-2025/blob/main/ejercicio-4_demo-selenium.py"

    %% Styles
    classDef workshopNode fill:#cce5ff,stroke:#333,stroke-width:2px;
    classDef exercise fill:#d0f0c0,stroke:#333,stroke-width:2px;
    classDef dependency fill:#ffe5b4,stroke:#333,stroke-width:2px;
    classDef extSystem fill:#f0e68c,stroke:#333,stroke-width:2px;
```
