# luminati-io/yandex-api architecture

> Architecture diagram of the GitHub repository luminati-io/yandex-api, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/luminati-io/yandex-api
- Repository: https://github.com/luminati-io/yandex-api

## Overview

##### Project Type & Purpose
- This project is a web scraping tool specialized in extracting search results from Yandex SERPs. It offers two alternatives:
- A free, basic scraper built in Python that leverages Playwright for browser automation and BeautifulSoup for HTML parsing.
- An enterprise-grade solution that integrates with Bright Data’s SERP Scraper API, using both direct API access and native proxy-based methods.
- The README clearly describes the use cases, limitations (e.g., challenges with CAPTCHAs), and detailed query parameters for tailoring searches, highlighting the project’s purpose to provide scalable real-time data extraction from Yandex search results.

##### File Structure Analysis
- Top-level files/folders include:
- README.md: Contains comprehensive project documentation including setup, usage, and technical details.
- images: Contains visual assets (screenshots for outputs and examples).
- setup-serp-api-guide.md: A guide to configuring the enterprise API version.
- yandex-scraper-api-output: Stores output HTML files from API requests.
- yandex-serp-scraper directory with the main Python script (yandex-serp-scraper.py) for the free scraper.

##### Core Components and Relationships
- Free Yandex SERP Scraper:
- Component: yandex-serp-scraper.py (Python script).
- Dependencies: Playwright (for controlling headless browser interactions) and BeautifulSoup (for parsing HTML content).
- Interaction: The script sends a browser-driven query to Yandex, receives the rendered page, and parses the HTML to extract data. The output is stored in yandex-scraper-api-output (for demonstration/archival purposes).
- Enterprise SERP API Integration:
- Component: API access scripts (illustrated by cURL and Python examples in the README).
- Dependencies: Python’s requests module, Bright Data’s API endpoints, and proxy infrastructure.
- Interaction: The script constructs a JSON payload (or uses proxy setup) and sends it to the Bright Data endpoint “https://api.brightdata.com/request”. The API returns results (raw HTML) which are saved as output.
- External Services and Infrastructure:
- Yandex Search Engine: The target data source.
- Bright Data’s SERP API & Proxy Infrastructure: Manages high-volume, real-time scraping for the enterprise approach.
- (Optional) SSL Certificates: Mentioned for secure proxy communication (emphasizing security best practices).

##### Important Architectural Patterns & Design Principles
- Separation of Concerns: Clear differentiation between the small-scale free scraper and the enterprise scalable solution.
- Modular Design: The free scraper and the enterprise API access operate independently within the repository.
- Use of External Services: Emphasis on leveraging established services (Bright Data API, proxy infrastructure) to handle scalability and anti-bot challenges.
- Configurability & Extensibility: Incorporation of parameters like query strings (pagination, localization, device targeting) hint at a design that is flexible and easy to extend.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Free Yandex SERP Scraper Flow
    subgraph "Free Yandex SERP Scraper Flow"
        FS["Free Scraper Module"]:::free
        PT["Playwright"]:::free
        BS["BeautifulSoup"]:::free
        YS["Yandex Search Engine"]:::external
    end

    %% Enterprise SERP API Flow
    subgraph "Enterprise SERP API Flow"
        EA["Enterprise API Client"]:::enterprise
        RP["Requests&Proxy"]:::enterprise
        BD["Bright Data SERP API"]:::external
        EG["Enterprise API Guide"]:::enterprise
    end

    %% Shared and Supporting Components
    OS(("Output Storage")):::storage
    Doc["Project Documentation"]:::documentation
    Note["Note:CAPTCHAs&IPblockingChallenges"]:::note

    %% Free Scraper Flow Connections
    FS -->|"uses"| PT
    FS -->|"uses"| BS
    PT -->|"controls"| YS
    YS -->|"returnsHTML"| BS
    BS -->|"outputsTo"| OS

    %% Enterprise API Flow Connections
    EG -->|"configures"| EA
    EA -->|"uses"| RP
    RP -->|"calls"| BD
    BD -->|"returnsHTML"| EA
    EA -->|"outputsTo"| OS

    %% Optional External Connection (API abstracts Yandex)
    BD -->|"abstracts"| YS

    %% Documentation and Note Placement
    Doc ---|"details"| FS
    Doc ---|"details"| EA
    Note --- FS
    Note --- EA

    %% Click Events
    click FS "https://github.com/luminati-io/yandex-api/blob/main/yandex-serp-scraper/yandex-serp-scraper.py"
    click EG "https://github.com/luminati-io/yandex-api/blob/main/setup-serp-api-guide.md"
    click OS "https://github.com/luminati-io/yandex-api/tree/main/yandex-scraper-api-output"
    click Doc "https://github.com/luminati-io/yandex-api/blob/main/README.md"

    %% Styles
    classDef free fill:#AED6F1,stroke:#1A5276,stroke-width:2px;
    classDef enterprise fill:#FAD7A0,stroke:#D35400,stroke-width:2px;
    classDef external fill:#C8E6C9,stroke:#388E3C,stroke-width:2px,stroke-dasharray: 5 5;
    classDef storage fill:#FFECB3,stroke:#F57F17,stroke-width:2px;
    classDef documentation fill:#E1BEE7,stroke:#6A1B9A,stroke-width:2px;
    classDef note fill:#EFEFEF,stroke:#888,stroke-dasharray: 5 5,stroke-width:2px;
```
