How sky-py/parse_shops works
Project Type and Purpose
• The project is a specialized parser for shop websites. Its goal is to extract product categories and detailed product information.
• The README explains that developers extend the base Parser class (from parser.py) by overriding three key methods (get_categories_links, get_products_links, get_product_info) to customize parsing for individual shop sites.
• The project supports asynchronous operations, optional JavaScript rendering via Playwright, and parameterization for handling discounts, excluded links, and more.
Analysis of the File Structure
• At the top level, you have multiple Python files named after specific shop domains (e.g., akri.com.ua.py, alvi-prague_ua.py). Each of these is likely a subclass of the base Parser that implements site-specific logic.
• The core functionality resides in files such as parser.py (containing base Parser and Product classes) and constants.py (for shared configuration).
• Supporting modules include messengers.py, retry.py, as well as modules for handling Excel outputs (xls2xlsx.py, xls_functions.py) which show how parsed data is processed and saved.
• There is also a requirements.txt to manage dependencies and GitHub workflow files for deployments, outlining integration and CI/CD practices.
a. Main Components:
• Core Parser Framework: represented by parser.py, which defines the Parser base class and Product entity.
• Site-Specific Parser Implementations: the various [site].py files (e.g., akri.com.ua.py, art of choice, etc.) extending the base parser.
• Utility and Configuration Modules: constants.py, messengers.py, and retry.py that support error handling, logging/messaging, and configuration.
• Data Export Layer: xls2xlsx.py and xls_functions.py that manage data conversion and Excel export tasks.
• External Dependencies: libraries like Playwright (installed as per the README), and any asynchronous frameworks used.
b. Relationships and Interactions:
• The core Parser class acts as the foundation. Site-specific parser files import and extend this class to implement custom logic.
• The Parser uses asynchronous workers to fetch pages, as defined by parameters (e.g., number_of_workers, worker_timeout) that control concurrency.
• Upon parsing, the product data (instances of Product) is processed and then passed to the data export functionality.
• Utility modules (for error handling, messaging, and retry logic) are used alongside the core and site-specific logic to enhance robustness.
c. Architectural Patterns & Design Principles:
• Inheritance/Extension: Custom parsers extend the base Parser to implement their own logic.
• Modular Design: Separates core parsing logic, site specific adaptations, and utility functions (error handling, Excel export) into distinct modules.
• Asynchronous Processing: Uses concurrent asynchronous workers to efficiently fetch and parse pages.
d. Incorporating Technologies and Frameworks:
• Use of Python for core logic, with asynchronous libraries (likely asyncio) for concurrency.
• Use of Playwright for handling JavaScript-rendered pages if render_javascript is enabled.
• That Excel conversion libraries and functions are key when handling output data.
Final Reminders
• Every component is clearly labeled, with annotations explaining its role.
• Details like parameter settings (e.g., number_of_workers, headless mode settings in Parser) where relevant.