# sagunsh/sofifa-scraper architecture

> Architecture diagram of the GitHub repository sagunsh/sofifa-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sagunsh/sofifa-scraper
- Repository: https://github.com/sagunsh/sofifa-scraper

## Overview

##### Main Components

a) Input/Configuration Layer:
- CLI Arguments Handler
- Configuration Parameters (filename, max_pages, year)

b) Core Scraping Engine:
- Web Scraper Component
- Data Parser
- Rate Limiter/Request Manager

c) Output Layer:
- File Writer (CSV/JSON)

##### Component Relationships

- User input → CLI Arguments Handler
- CLI Handler → Scraper Configuration
- Scraper → External Service (sofifa.com)
- Scraper → Data Parser
- Parser → File Writer

- [User Input] → [CLI Arguments Parser]
- ↓
- [Configuration] → [Rate Limiter] → [Web Scraper] ← [sofifa.com]
- ↓
- [Data Parser]
- ↓
- [File Writer] → [Output File]

##### Additional Details

- Supported file formats (CSV/JSON) in the File Writer component
- HTTP requests flow to sofifa.com
- A note about rate limiting/responsible scraping
- Version compatibility (Python 3.6+)

##### Technical Specifications to Note

- Key Python dependencies from requirements.txt
- Command-line interface options
- Error handling paths

##### Important Notes

- A note about responsible usage
- Error handling paths
- Where rate limiting occurs

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Legend
        style Legend fill:#f5f5f5,stroke:#333
        L1[Component]:::core
        L2[/Input/Output/]:::io
        L3[(External Service)]:::external
        L4[[Processing Unit]]:::process
        L5[Configuration]:::config
    end

    subgraph Input
        UserInput[/User Input/]:::io
        CLI[[CLI Arguments Parser]]:::process
        Config[Configuration]:::config
    end

    subgraph Core["Core Scraping Engine"]
        RateLimiter[[Rate Limiter]]:::process
        WebScraper[[Web Scraper]]:::process
        DataParser[[Data Parser]]:::process
    end

    subgraph External
        SoFIFA[(sofifa.com)]:::external
    end

    subgraph Output
        FileWriter[[File Writer]]:::process
        OutputFile[/Output File<br>CSV/JSON/]:::io
    end

    UserInput --> CLI
    CLI --> Config
    Config --> RateLimiter
    RateLimiter --> WebScraper
    WebScraper <--> SoFIFA
    WebScraper --> DataParser
    DataParser --> FileWriter
    FileWriter --> OutputFile

    %% Component click events
    click CLI "https://github.com/sagunsh/sofifa-scraper/blob/master/scrape_sofifa.py"
    click RateLimiter "https://github.com/sagunsh/sofifa-scraper/blob/master/scrape_sofifa.py"
    click WebScraper "https://github.com/sagunsh/sofifa-scraper/blob/master/scrape_sofifa.py"
    click DataParser "https://github.com/sagunsh/sofifa-scraper/blob/master/scrape_sofifa.py"
    click FileWriter "https://github.com/sagunsh/sofifa-scraper/blob/master/scrape_sofifa.py"
    click Config "https://github.com/sagunsh/sofifa-scraper/blob/master/requirements.txt"

    %% Styling
    classDef core fill:#ddf,stroke:#333
    classDef io fill:#9f9,stroke:#333
    classDef external fill:#eee,stroke:#333
    classDef process fill:#ffd,stroke:#333
    classDef config fill:#ff9,stroke:#333

    %% Notes
    note["Note: Implements responsible<br>scraping with rate limiting"]
    style note fill:#fff,stroke:#333
```
