# sky-py/parse_shops architecture

> Architecture diagram of the GitHub repository sky-py/parse_shops, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sky-py/parse_shops
- Repository: https://github.com/sky-py/parse_shops

## Overview

##### Project Type and Purpose
- The project is a specialized parser for shop websites. Its goal is to extract product categories and detailed product information.
- The README explains that developers extend the base Parser class (from parser.py) by overriding three key methods (get_categories_links, get_products_links, get_product_info) to customize parsing for individual shop sites.
- The project supports asynchronous operations, optional JavaScript rendering via Playwright, and parameterization for handling discounts, excluded links, and more.

##### Analysis of the File Structure
- At the top level, you have multiple Python files named after specific shop domains (e.g., akri.com.ua.py, alvi-prague_ua.py). Each of these is likely a subclass of the base Parser that implements site-specific logic.
- The core functionality resides in files such as parser.py (containing base Parser and Product classes) and constants.py (for shared configuration).
- Supporting modules include messengers.py, retry.py, as well as modules for handling Excel outputs (xls2xlsx.py, xls_functions.py) which show how parsed data is processed and saved.
- There is also a requirements.txt to manage dependencies and GitHub workflow files for deployments, outlining integration and CI/CD practices.

a. Main Components:
- Core Parser Framework: represented by parser.py, which defines the Parser base class and Product entity.
- Site-Specific Parser Implementations: the various [site].py files (e.g., akri.com.ua.py, art of choice, etc.) extending the base parser.
- Utility and Configuration Modules: constants.py, messengers.py, and retry.py that support error handling, logging/messaging, and configuration.
- Data Export Layer: xls2xlsx.py and xls_functions.py that manage data conversion and Excel export tasks.
- External Dependencies: libraries like Playwright (installed as per the README), and any asynchronous frameworks used.

b. Relationships and Interactions:
- The core Parser class acts as the foundation. Site-specific parser files import and extend this class to implement custom logic.
- The Parser uses asynchronous workers to fetch pages, as defined by parameters (e.g., number_of_workers, worker_timeout) that control concurrency.
- Upon parsing, the product data (instances of Product) is processed and then passed to the data export functionality.
- Utility modules (for error handling, messaging, and retry logic) are used alongside the core and site-specific logic to enhance robustness.

c. Architectural Patterns & Design Principles:
- Inheritance/Extension: Custom parsers extend the base Parser to implement their own logic.
- Modular Design: Separates core parsing logic, site specific adaptations, and utility functions (error handling, Excel export) into distinct modules.
- Asynchronous Processing: Uses concurrent asynchronous workers to efficiently fetch and parse pages.

d. Incorporating Technologies and Frameworks:
- Use of Python for core logic, with asynchronous libraries (likely asyncio) for concurrency.
- Use of Playwright for handling JavaScript-rendered pages if render_javascript is enabled.
- That Excel conversion libraries and functions are key when handling output data.

##### Final Reminders
- Every component is clearly labeled, with annotations explaining its role.
- Details like parameter settings (e.g., number_of_workers, headless mode settings in Parser) where relevant.

## Diagram (Mermaid)

```mermaid
flowchart TD
    A["Initiate Parsing (Parser().parse())"]:::start
    A --> B["Core Parser Framework (parser.py)"]:::core

    subgraph "Parsing Process"
        C["Extract Category Links (get_categories_links)"]:::process
        D["Extract Product Links (get_products_links)"]:::process
        E["Extract Product Info (get_product_info)"]:::process
        F["Process & Export Data"]:::export
        B --> C
        C --> D
        D --> E
        E --> F
    end

    subgraph "Site-specific Parser Modules"
        G["akri.com.ua.py"]:::site
        H["alvi-prague_ua.py"]:::site
        I["artofchoice.ua.py"]:::site
        J["beauty-service.py"]:::site
        K["gsdm.com.ua.py"]:::site
        L["home-club.com.ua.py"]:::site
        M["ramosu.com.ua.py"]:::site
        N["sunuv.in.ua.py"]:::site
        O["teri.ua.py"]:::site
    end

    G ---|"extends"| B
    H ---|"extends"| B
    I ---|"extends"| B
    J ---|"extends"| B
    K ---|"extends"| B
    L ---|"extends"| B
    M ---|"extends"| B
    N ---|"extends"| B
    O ---|"extends"| B

    subgraph "Utility & Configuration Modules"
        P["Configuration (constants.py)"]:::config
        Q["Messenger (messengers.py)"]:::util
        R["Retry Logic (retry.py)"]:::util
    end

    B -->|"uses"| P
    B -->|"uses"| Q
    B -->|"uses"| R

    subgraph "External Libraries & Async"
        S["Async Workers & Libraries (asyncio, Playwright)"]:::external
    end

    B -->|"handles async via"| S

    subgraph "Data Export Modules"
        T["Excel Converter (xls2xlsx.py)"]:::export
        U["Excel Functions (xls_functions.py)"]:::export
    end

    F -->|"exports data to"| T
    F -->|"exports data to"| U

    click B "https://github.com/sky-py/parse_shops/blob/master/parser.py"
    click G "https://github.com/sky-py/parse_shops/blob/master/akri.com.ua.py"
    click H "https://github.com/sky-py/parse_shops/blob/master/alvi-prague_ua.py"
    click I "https://github.com/sky-py/parse_shops/blob/master/artofchoice.ua.py"
    click J "https://github.com/sky-py/parse_shops/blob/master/beauty-service.py"
    click K "https://github.com/sky-py/parse_shops/blob/master/gsdm.com.ua.py"
    click L "https://github.com/sky-py/parse_shops/blob/master/home-club.com.ua.py"
    click M "https://github.com/sky-py/parse_shops/blob/master/ramosu.com.ua.py"
    click N "https://github.com/sky-py/parse_shops/blob/master/sunuv.in.ua.py"
    click O "https://github.com/sky-py/parse_shops/blob/master/teri.ua.py"
    click P "https://github.com/sky-py/parse_shops/blob/master/constants.py"
    click Q "https://github.com/sky-py/parse_shops/blob/master/messengers.py"
    click R "https://github.com/sky-py/parse_shops/blob/master/retry.py"
    click T "https://github.com/sky-py/parse_shops/blob/master/xls2xlsx.py"
    click U "https://github.com/sky-py/parse_shops/blob/master/xls_functions.py"

    classDef start fill:#f0f9ff,stroke:#1f78b4,stroke-width:2px;
    classDef core fill:#cce5ff,stroke:#1f78b4,stroke-width:2px;
    classDef process fill:#d1e7dd,stroke:#0f5132,stroke-width:2px;
    classDef export fill:#f8d7da,stroke:#842029,stroke-width:2px;
    classDef site fill:#e2e3e5,stroke:#41464b,stroke-width:2px;
    classDef config fill:#fff3cd,stroke:#664d03,stroke-width:2px;
    classDef util fill:#ffe5b4,stroke:#d48806,stroke-width:2px;
    classDef external fill:#dbe9f4,stroke:#31708f,stroke-width:2px;
```
