# m3mons/hltv-scrapper-api architecture

> Architecture diagram of the GitHub repository m3mons/hltv-scrapper-api, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/m3mons/hltv-scrapper-api
- Repository: https://github.com/m3mons/hltv-scrapper-api

## Overview

##### Main Components
- Client/API Consumer: Represent external users (or applications) consuming the API endpoints (e.g., /teams/top, /team/<name>, /results, /matches/upcoming).
- Flask Web Server (app.py): This is the API entry point that exposes the routes indicated in the README. It handles HTTP requests and returns JSON responses.
- Business Logic Layer (“classes” directory): This layer comprises various helper classes (e.g., Cleaner, Conditions, ConditionsChecker, Data, Process, SpiderManager, etc.) which encapsulate data processing, validation, and orchestration logic. These classes coordinate with both incoming HTTP requests and the scraping operations.
- Scraping Subsystem (hltv_scraper directory): Built using Scrapy, this subsystem includes:
- The main Scrapy project (the hltv_scraper/hltv_scraper package),
- Multiple spiders (located in the spiders subdirectory) targeted at different endpoints (teams, matches, news, etc.),
- Parsers (in the parsers subdirectory) that extract and structure the raw data.
- External Data Source: The hltv.org website from which live data is being scraped.

##### Relationships and Interactions
- HTTP Flow: The client sends HTTP requests to the Flask API. The Flask app (app.py) parses these requests and invokes the relevant business logic methods.
- Business Logic Coordination: The classes (e.g., SpiderManager) manage and initiate scraping tasks by interfacing with the Scrapy subsystem, providing necessary inputs (like URL paths, conditions, or parameters) and handling output data.
- Scraping Process: Within the hltv_scraper, specific spiders target endpoints on hltv.org. The spiders use dedicated parsers to extract valuable information. This data is either directly delivered back to the Flask API via the business logic layer or pipelined through Scrapy’s built-in mechanisms.
- Data Flow: Data flows from the external data source (hltv.org) through the Scrapy spiders and parsers, then gets processed by the business logic (cleaning, validating, formatting) before being exposed back through the Flask API to the client.

##### Architectural Patterns & Design Principles
- Separation of Concerns: The API is kept distinct from the data scraping logic — the Flask layer only exposes endpoints and delegates data retrieval and processing to the classes and Scrapy spiders.
- Modular Design: With separate directories for “classes” (business logic) and “hltv_scraper” (data retrieval), the project makes it straightforward to update or extend each component without affecting the other.
- Pipeline and Factory Patterns: Noticeable in the “classes” (such as conditions_factory) and within the Scrapy pipelines, ensuring that data is processed in stages and can be extended seamlessly.

## Diagram (Mermaid)

```mermaid
graph TD
    Client("Client/API Consumer"):::client
    API("Flask API (app.py)"):::flask
    Logic("Business Logic (classes)"):::business
    Scrapy("Scrapy Project (hltv_scraper/hltv_scraper)"):::scrapy
    Spiders("Spiders (hltv_scraper/hltv_scraper/spiders)"):::scrapy
    Parsers("Parsers (hltv_scraper/hltv_scraper/spiders/parsers)"):::scrapy
    External("hltv.org (External Data Source)"):::external

    %% Request Flow
    Client -->|"HTTP_Request"| API
    API -->|"Delegate_Request"| Logic
    Logic -->|"Invoke_Scraping"| Scrapy
    Scrapy -->|"Trigger_Spiders"| Spiders
    Spiders -->|"Utilize_Parsers"| Parsers
    Parsers -->|"Extract_Data"| External

    %% Response Flow
    External -->|"Data_Response"| Parsers
    Parsers -->|"Parsed_Data"| Spiders
    Spiders -->|"Scraped_Output"| Scrapy
    Scrapy -->|"Scraping_Result"| Logic
    Logic -->|"API_Response"| API
    API -->|"HTTP_Response"| Client

    %% Click Events
    click API "https://github.com/m3mons/hltv-scrapper-api/blob/main/app.py"
    click Logic "https://github.com/m3mons/hltv-scrapper-api/tree/main/classes/"
    click Scrapy "https://github.com/m3mons/hltv-scrapper-api/tree/main/hltv_scraper/hltv_scraper/"
    click Spiders "https://github.com/m3mons/hltv-scrapper-api/tree/main/hltv_scraper/hltv_scraper/spiders/"
    click Parsers "https://github.com/m3mons/hltv-scrapper-api/tree/main/hltv_scraper/hltv_scraper/spiders/parsers/"

    %% Styles
    classDef client fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef flask fill:#F5B041,stroke:#D35400,stroke-width:2px;
    classDef business fill:#FCF3CF,stroke:#B7950B,stroke-width:2px;
    classDef scrapy fill:#82E0AA,stroke:#27AE60,stroke-width:2px;
    classDef external fill:#C39BD3,stroke:#8E44AD,stroke-width:2px;
```
