# zukelima/abicom-webscrapping-project architecture

> Architecture diagram of the GitHub repository zukelima/abicom-webscrapping-project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zukelima/abicom-webscrapping-project
- Repository: https://github.com/zukelima/abicom-webscrapping-project

## Overview

##### Project Type and Purpose
- This project is a domain-specific scraper and image analysis tool. Its primary purpose is to automate the process of scraping images of fuel price reports from the Abicom website and then running an advanced analysis on these images to extract structured data (including tabular data via OCR and layout analysis).
- The project integrates web scraping, image metadata extraction (using Pillow), OCR and table extraction (using img2table and easyocr), and data consolidation (using pandas).

##### File Structure
- Major directories include:
- src/ – Contains the main application code organized into modules.
- src/scrapers/ – Houses base and specialized scraper classes (e.g., abicom_scraper.py) for fetching images from the website.
- src/services/ – Provides auxiliary services; for instance, the HTTP client for robust network requests and the ImageService to avoid duplicate downloads.
- src/utils/ – Contains utility functions such as file and URL handling.
- src/models/ – Contains data representations (like the image model).
- data/ – Serves as the storage location for downloaded images and the CSV reports generated after image processing.
- Auxiliary directories (.devcontainer, .vscode, .idea) indicate support for containerized and IDE-based development environments but are not part of the core business logic.

##### README for Additional Insights
- The README details two main operational pipelines:
a. The web scraping pipeline (initiated via src/main.py) is responsible for downloading images from the Abicom website.
b. The image analysis pipeline (triggered via src/analise_imagens.py) runs parallel processing on the images to extract metadata and perform OCR-driven table extraction.
- It also highlights important features and design choices like the use of concurrent processing to speed up analysis, the usage of a custom HTTP client with retry logic, and the extraction and refinement of numerical data values from tables.

a. Main Components:
- Web Scraping Module:
- Entry point: src/main.py
- Scrapers: src/scrapers/base_scraper.py and src/scrapers/abicom_scraper.py
- Services: src/services/http_client.py for robust fetching and src/services/image_service.py for handling image storage and avoiding duplicates.
- Image Analysis Module:
- Main processing: src/analise_imagens.py
- Uses: Pillow for image properties, img2table and easyocr for OCR and table extraction.
- Data Storage:
- Local file system (data/images for images; data/*.csv for analysis reports).
- Utility Components:
- Configurations: src/config.py, which sets parameters such as output directories and pagination limits.
- Utils: src/utils (file_utils.py, url_utils.py) to support file and URL manipulations.
- Development/Deployment Aids:
- Docker configuration (.devcontainer and docker-compose.yml) to facilitate containerized development and deployment.
- Logging: The presence of a scraper.log file indicates robust logging for debugging and monitoring.

b. Relationships and Interactions:
- A separate branch depicting the image analysis where images from the data/images directory flow into the analysis module which processes them via OCR and table extraction.
- All processed data then consolidates into CSV outputs, which should be depicted as the final output storage.
- Interaction with external systems like the Abicom website and dependencies such as the OCR and computer vision libraries.

c. Architectural Patterns and Design Principles:
- Modularity: Separation between scraping, service, and analysis modules.
- Concurrency: Notation of parallel processing (concurrent.futures.ProcessPoolExecutor) in the image analysis workflow.
- Robustness & Extensibility: Use of custom HTTP clients with retry logic, pre-indexing in the ImageService, and a clearly defined configuration module.
- Layering: Clear division between core business logic (scrapers, analysis) and supporting utilities (utils, models).

d. Relevant Technologies and Frameworks:
- Python 3.8+ as the core language.
- Web scraping libraries: requests and BeautifulSoup.
- Image processing: Pillow.
- OCR and table extraction: easyocr and img2table, plus relevant deep learning backends (torch).
- Data manipulation: pandas and numpy.
- Support for containerization: Docker and dev container configuration files.

##### Implementation Steps for the Principal Software Engineer
- Branch out to show the web scraping part (linking to the scrapers and services) and note the dependencies on external HTTP services.
- Incorporate support elements (configuration, logging, container setup) in a sidebar or a “supporting services” section.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Main Orchestrator
    A["Main Orchestrator (src/main.py)"]:::orchestrator

    %% Web Scraping Pipeline Subgraph
    subgraph "Web Scraping Pipeline"
        B["Base Scraper (src/scrapers/base_scraper.py)"]:::scraper
        C["Abicom Scraper (src/scrapers/abicom_scraper.py)"]:::scraper
        D["HTTP Client Service (src/services/http_client.py)"]:::service
        E["Image Service (src/services/image_service.py)"]:::service
    end

    %% Data Storage Node
    F["Data Storage (data/)"]:::storage

    %% Image Analysis Pipeline Subgraph
    subgraph "Image Analysis Pipeline"
        G["Image Analysis Module (src/analise_imagens.py)"]:::analysis
        O["OCR & Table Extraction Libraries"]:::external
    end

    %% Supporting Utilities Subgraph
    subgraph "Supporting Utilities"
        H["Configuration (src/config.py)"]:::util
        I["File Utils (src/utils/file_utils.py)"]:::util
        J["URL Utils (src/utils/url_utils.py)"]:::util
        K["Image Data Model (src/models/image.py)"]:::util
    end

    %% Development/Deployment Aids Subgraph
    subgraph "Dev/Deployment Aids"
        L["Dev Container (.devcontainer)"]:::dev
        M["Docker Compose (docker-compose.yml)"]:::dev
    end

    %% External Entities
    N["Abicom Website"]:::external

    %% Connections from Main Orchestrator
    A -->|"triggers_scraping"| B
    A -->|"triggers_scraping"| C
    A -->|"triggers_analysis"| G
    A -->|"loads_config"| H

    %% Web Scraping Module Interactions
    B -->|"uses_HTTP"| D
    C -->|"uses_HTTP"| D
    C -->|"invokes_storage"| E
    E -->|"stores_images"| F
    N -->|"provides_web_content"| C

    %% Image Analysis Pipeline Interactions
    F -->|"supplies_images"| G
    G -->|"outputs_CSV_to"| F
    G -->|"utilizes"| O

    %% Supporting Utilities Relationships
    H --> I
    H --> J
    H --> K

    %% Styles
    classDef orchestrator fill:#f9f,stroke:#333,stroke-width:2px;
    classDef service fill:#ccf,stroke:#333,stroke-width:2px;
    classDef scraper fill:#cfc,stroke:#333,stroke-width:2px;
    classDef analysis fill:#fcf,stroke:#333,stroke-width:2px;
    classDef util fill:#cff,stroke:#333,stroke-width:2px;
    classDef storage fill:#ffc,stroke:#333,stroke-width:2px;
    classDef dev fill:#ddd,stroke:#333,stroke-width:2px;
    classDef external fill:#efe,stroke:#333,stroke-width:2px;

    %% Click Events
    click A "https://github.com/zukelima/abicom-webscrapping-project/blob/main/src/main.py"
    click B "https://github.com/zukelima/abicom-webscrapping-project/blob/main/src/scrapers/base_scraper.py"
    click C "https://github.com/zukelima/abicom-webscrapping-project/blob/main/src/scrapers/abicom_scraper.py"
    click D "https://github.com/zukelima/abicom-webscrapping-project/blob/main/src/services/http_client.py"
    click E "https://github.com/zukelima/abicom-webscrapping-project/blob/main/src/services/image_service.py"
    click G "https://github.com/zukelima/abicom-webscrapping-project/blob/main/src/analise_imagens.py"
    click H "https://github.com/zukelima/abicom-webscrapping-project/blob/main/src/config.py"
    click I "https://github.com/zukelima/abicom-webscrapping-project/blob/main/src/utils/file_utils.py"
    click J "https://github.com/zukelima/abicom-webscrapping-project/blob/main/src/utils/url_utils.py"
    click K "https://github.com/zukelima/abicom-webscrapping-project/blob/main/src/models/image.py"
    click F "https://github.com/zukelima/abicom-webscrapping-project/tree/main/data/"
    click L "https://github.com/zukelima/abicom-webscrapping-project/blob/main/.devcontainer"
    click M "https://github.com/zukelima/abicom-webscrapping-project/blob/main/docker-compose.yml"
```
