# carlosabrizuela/scraper_hlibertad_requests architecture

> Architecture diagram of the GitHub repository carlosabrizuela/scraper_hlibertad_requests, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/carlosabrizuela/scraper_hlibertad_requests
- Repository: https://github.com/carlosabrizuela/scraper_hlibertad_requests

## Overview

##### Main Components
- Entry Point – main.py: This is the central script that orchestrates the overall process.
- Configuration Loader – files/config.yaml: Configuration settings (API URLs, proxy settings, threading parameters, pagination limits, and output directory) are loaded via PyYAML.
- Scraper Module – Scraper.py: Contains the core scraping logic (making HTTP requests to fetch categories and product data, constructing search URLs, and handling pagination).
- Utility Functions – utility_functions.py: Implements helper functions that may include URL formatting, error handling, and miscellaneous tasks.
- External Dependencies (Libraries): Requests (for HTTP calls), PyYAML (for configuration), and Pandas (for generating CSV files).
- External Services: VTEX API endpoints (for categories and product search), which the scraper interacts with.
- Multithreading Component: Although implicit, the configuration includes a thread_number setting, indicating that the scraper might use concurrent threads for efficiency.
- Output Component: Generates CSV files (each named in the format [date]__[category-name].csv) in the specified output directory.

##### Relationships and Interactions
- main.py reads the configuration (via PyYAML from config.yaml) and then calls into Scraper.py.
- Scraper.py utilizes utility_functions.py where necessary and uses the Requests library to query external VTEX API endpoints.
- Data flow: The scraper first fetches the list of categories, then iterates through each category to build search URLs with pagination. Each API result (JSON response) is processed (or possibly aggregated) and passed into Pandas for CSV file generation.
- The program may use multi-threading (as suggested by the “thread_number” in the config) to improve scraping performance, where each thread potentially handles separate API calls or pagination.
- CSV generation: Processed data flows into Pandas for formatting and output, saving to the designated directory.

##### Architectural Patterns and Design Principles
- Separation of Concerns: The code is modular—configuration is isolated from business logic (Scraper.py) and helper functions (utility_functions.py).
- Layered Structure:
- Configuration and Initialization Layer (main.py reading config.yaml).
- Business Logic Layer (Scraper.py coordinating scraping and data processing).
- Utility Layer (utility_functions.py provides supplementary support).
- Data Output Layer (using Pandas to write CSV files).
- Concurrency: The usage of thread_number suggests that the project might leverage multi-threading for parallel HTTP requests.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Initialization Layer"
        MAIN["main.py (Entry Point)"]:::entry
        CONFIG["Config Loader (config.yaml)"]:::config
    end

    subgraph "Scraping Process"
        SCRAPER["Scraper Module (Scraper.py)"]:::scraper
        UTILITY["Utility Functions (utility_functions.py)"]:::utility
        VTEX_CAT["VTEX API\n(Category Endpoint)"]:::external
        VTEX_SEARCH["VTEX API\n(Product Search Endpoint)"]:::external
        CSV_GEN["CSV Generation\n(Pandas)"]:::csv
        THREAD_POOL["Multithreading Component"]:::threads
    end

    MAIN -->|"reads_config"| CONFIG
    CONFIG -->|"initializes_scraper"| SCRAPER
    SCRAPER -->|"calls_helpers"| UTILITY
    SCRAPER -->|"fetches_categories"| VTEX_CAT
    SCRAPER -->|"fetches_products"| VTEX_SEARCH
    VTEX_CAT -->|"returns_data"| SCRAPER
    VTEX_SEARCH -->|"returns_data"| SCRAPER
    SCRAPER -->|"processes_data"| CSV_GEN
    SCRAPER -->|"utilizes_concurrency"| THREAD_POOL

    click MAIN "https://github.com/carlosabrizuela/scraper_hlibertad_requests/blob/main/main.py"
    click CONFIG "https://github.com/carlosabrizuela/scraper_hlibertad_requests/blob/main/files/config.yaml"
    click SCRAPER "https://github.com/carlosabrizuela/scraper_hlibertad_requests/blob/main/Scraper.py"
    click UTILITY "https://github.com/carlosabrizuela/scraper_hlibertad_requests/blob/main/utility_functions.py"

    classDef entry fill:#a0c4ff,stroke:#333,stroke-width:2px;
    classDef config fill:#ffd6a5,stroke:#333,stroke-width:2px;
    classDef scraper fill:#ffadad,stroke:#333,stroke-width:2px;
    classDef utility fill:#caffbf,stroke:#333,stroke-width:2px;
    classDef external fill:#fdffb6,stroke:#333,stroke-width:2px;
    classDef csv fill:#baffc9,stroke:#333,stroke-width:2px;
    classDef threads fill:#ffb3c6,stroke:#333,stroke-width:2px;
```
