How carlosabrizuela/scraper_hlibertad_requests works
Main Components
• Entry Point – main.py: This is the central script that orchestrates the overall process.
• Configuration Loader – files/config.yaml: Configuration settings (API URLs, proxy settings, threading parameters, pagination limits, and output directory) are loaded via PyYAML.
• Scraper Module – Scraper.py: Contains the core scraping logic (making HTTP requests to fetch categories and product data, constructing search URLs, and handling pagination).
• Utility Functions – utility_functions.py: Implements helper functions that may include URL formatting, error handling, and miscellaneous tasks.
• External Dependencies (Libraries): Requests (for HTTP calls), PyYAML (for configuration), and Pandas (for generating CSV files).
• External Services: VTEX API endpoints (for categories and product search), which the scraper interacts with.
• Multithreading Component: Although implicit, the configuration includes a thread_number setting, indicating that the scraper might use concurrent threads for efficiency.
• Output Component: Generates CSV files (each named in the format [date]__[category-name].csv) in the specified output directory.
Relationships and Interactions
• main.py reads the configuration (via PyYAML from config.yaml) and then calls into Scraper.py.
• Scraper.py utilizes utility_functions.py where necessary and uses the Requests library to query external VTEX API endpoints.
• Data flow: The scraper first fetches the list of categories, then iterates through each category to build search URLs with pagination. Each API result (JSON response) is processed (or possibly aggregated) and passed into Pandas for CSV file generation.
• The program may use multi-threading (as suggested by the “thread_number” in the config) to improve scraping performance, where each thread potentially handles separate API calls or pagination.
• CSV generation: Processed data flows into Pandas for formatting and output, saving to the designated directory.
Architectural Patterns and Design Principles
• Separation of Concerns: The code is modular—configuration is isolated from business logic (Scraper.py) and helper functions (utility_functions.py).
• Layered Structure:
• Configuration and Initialization Layer (main.py reading config.yaml).
• Business Logic Layer (Scraper.py coordinating scraping and data processing).
• Utility Layer (utility_functions.py provides supplementary support).
• Data Output Layer (using Pandas to write CSV files).
• Concurrency: The usage of thread_number suggests that the project might leverage multi-threading for parallel HTTP requests.