How m3mons/hltv-scrapper-api works
Main Components
• Client/API Consumer: Represent external users (or applications) consuming the API endpoints (e.g., /teams/top, /team/<name>, /results, /matches/upcoming).
• Flask Web Server (app.py): This is the API entry point that exposes the routes indicated in the README. It handles HTTP requests and returns JSON responses.
• Business Logic Layer (“classes” directory): This layer comprises various helper classes (e.g., Cleaner, Conditions, ConditionsChecker, Data, Process, SpiderManager, etc.) which encapsulate data processing, validation, and orchestration logic. These classes coordinate with both incoming HTTP requests and the scraping operations.
• Scraping Subsystem (hltv_scraper directory): Built using Scrapy, this subsystem includes:
• The main Scrapy project (the hltv_scraper/hltv_scraper package),
• Multiple spiders (located in the spiders subdirectory) targeted at different endpoints (teams, matches, news, etc.),
• Parsers (in the parsers subdirectory) that extract and structure the raw data.
• External Data Source: The hltv.org website from which live data is being scraped.
Relationships and Interactions
• HTTP Flow: The client sends HTTP requests to the Flask API. The Flask app (app.py) parses these requests and invokes the relevant business logic methods.
• Business Logic Coordination: The classes (e.g., SpiderManager) manage and initiate scraping tasks by interfacing with the Scrapy subsystem, providing necessary inputs (like URL paths, conditions, or parameters) and handling output data.
• Scraping Process: Within the hltv_scraper, specific spiders target endpoints on hltv.org. The spiders use dedicated parsers to extract valuable information. This data is either directly delivered back to the Flask API via the business logic layer or pipelined through Scrapy’s built-in mechanisms.
• Data Flow: Data flows from the external data source (hltv.org) through the Scrapy spiders and parsers, then gets processed by the business logic (cleaning, validating, formatting) before being exposed back through the Flask API to the client.
Architectural Patterns & Design Principles
• Separation of Concerns: The API is kept distinct from the data scraping logic — the Flask layer only exposes endpoints and delegates data retrieval and processing to the classes and Scrapy spiders.
• Modular Design: With separate directories for “classes” (business logic) and “hltv_scraper” (data retrieval), the project makes it straightforward to update or extend each component without affecting the other.
• Pipeline and Factory Patterns: Noticeable in the “classes” (such as conditions_factory) and within the Scrapy pipelines, ensuring that data is processed in stages and can be extended seamlessly.