How zukelima/abicom-webscrapping-project works
Project Type and Purpose
• This project is a domain-specific scraper and image analysis tool. Its primary purpose is to automate the process of scraping images of fuel price reports from the Abicom website and then running an advanced analysis on these images to extract structured data (including tabular data via OCR and layout analysis).
• The project integrates web scraping, image metadata extraction (using Pillow), OCR and table extraction (using img2table and easyocr), and data consolidation (using pandas).
File Structure
• Major directories include:
• src/ – Contains the main application code organized into modules.
• src/scrapers/ – Houses base and specialized scraper classes (e.g., abicom_scraper.py) for fetching images from the website.
• src/services/ – Provides auxiliary services; for instance, the HTTP client for robust network requests and the ImageService to avoid duplicate downloads.
• src/utils/ – Contains utility functions such as file and URL handling.
• src/models/ – Contains data representations (like the image model).
• data/ – Serves as the storage location for downloaded images and the CSV reports generated after image processing.
• Auxiliary directories (.devcontainer, .vscode, .idea) indicate support for containerized and IDE-based development environments but are not part of the core business logic.
README for Additional Insights
• The README details two main operational pipelines:
a. The web scraping pipeline (initiated via src/main.py) is responsible for downloading images from the Abicom website.
b. The image analysis pipeline (triggered via src/analise_imagens.py) runs parallel processing on the images to extract metadata and perform OCR-driven table extraction.
• It also highlights important features and design choices like the use of concurrent processing to speed up analysis, the usage of a custom HTTP client with retry logic, and the extraction and refinement of numerical data values from tables.
a. Main Components:
• Web Scraping Module:
• Entry point: src/main.py
• Scrapers: src/scrapers/base_scraper.py and src/scrapers/abicom_scraper.py
• Services: src/services/http_client.py for robust fetching and src/services/image_service.py for handling image storage and avoiding duplicates.
• Image Analysis Module:
• Main processing: src/analise_imagens.py
• Uses: Pillow for image properties, img2table and easyocr for OCR and table extraction.
• Data Storage:
• Local file system (data/images for images; data/*.csv for analysis reports).
• Utility Components:
• Configurations: src/config.py, which sets parameters such as output directories and pagination limits.
• Utils: src/utils (file_utils.py, url_utils.py) to support file and URL manipulations.
• Development/Deployment Aids:
• Docker configuration (.devcontainer and docker-compose.yml) to facilitate containerized development and deployment.
• Logging: The presence of a scraper.log file indicates robust logging for debugging and monitoring.
b. Relationships and Interactions:
• A separate branch depicting the image analysis where images from the data/images directory flow into the analysis module which processes them via OCR and table extraction.
• All processed data then consolidates into CSV outputs, which should be depicted as the final output storage.
• Interaction with external systems like the Abicom website and dependencies such as the OCR and computer vision libraries.
c. Architectural Patterns and Design Principles:
• Modularity: Separation between scraping, service, and analysis modules.
• Concurrency: Notation of parallel processing (concurrent.futures.ProcessPoolExecutor) in the image analysis workflow.
• Robustness & Extensibility: Use of custom HTTP clients with retry logic, pre-indexing in the ImageService, and a clearly defined configuration module.
• Layering: Clear division between core business logic (scrapers, analysis) and supporting utilities (utils, models).
d. Relevant Technologies and Frameworks:
• Python 3.8+ as the core language.
• Web scraping libraries: requests and BeautifulSoup.
• Image processing: Pillow.
• OCR and table extraction: easyocr and img2table, plus relevant deep learning backends (torch).
• Data manipulation: pandas and numpy.
• Support for containerization: Docker and dev container configuration files.
Implementation Steps for the Principal Software Engineer
• Branch out to show the web scraping part (linking to the scrapers and services) and note the dependencies on external HTTP services.
• Incorporate support elements (configuration, logging, container setup) in a sidebar or a “supporting services” section.