# paperswithcode/sota-extractor architecture

> Architecture diagram of the GitHub repository paperswithcode/sota-extractor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/paperswithcode/sota-extractor
- Repository: https://github.com/paperswithcode/sota-extractor

## Overview

##### Main Components
- Command Line Interface (CLI) & Entry Points:
- The main CLI logic (located in sota_extractor/__main__.py and sota_extractor/commands/cli.py) is the entry point, handling user commands to run scrapers or evaluate data.
- Scraper Modules:
- A dedicated “scrapers” module (sota_extractor/scrapers) contains individual scraper scripts (e.g., chexpert.py, cityscapes.py, cmrc.py, etc.) for each external data source. Each scraper encapsulates logic to fetch and parse data from its respective website.
- Notice also a nested scraper for “nlp_progress”, which has its own substructure.
- Data Handling & Serialization:
- The “serialization.py” file manages the conversion of raw data into the structured JSON format described in the README.
- The “taskdb” sub-package (sota_extractor/taskdb/v01) provides models and schemas to represent the core entities (Task, Dataset, Sota, etc.) in a consistent way.
- Data Storage:
- The “data” folder holds both the static scraped outputs and the example JSON files (including tasks and metadata) that follow a defined structure.
- Evaluation Components:
- Commands like “evaluate.py” and references to the “eval_all_report.csv” in the README indicate an evaluation pipeline to check the performance of the extraction process.
- Utility & Error Handling:
- Utility modules (like “errors.py” and “utils.py”) are used to manage exceptions and common functions across the project.

##### Relationships & Interactions
- The CLI acts as a gateway to trigger the scrapers and evaluation routines.
- When a scraping command is invoked (e.g., python -m scrapers.squad), the corresponding scraper module initiates a request to the external service, processes the response and then serializes the result into the JSON format.
- The serialization logic interacts with the “taskdb” module to ensure data consistency (mapping tasks, datasets, and SOTA results).
- All resulting data is stored in the “data” directory.
- Evaluation commands pull the stored data (or freshly scraped data) and generate performance reports, such as the “eval_all_report.csv”.

##### Architectural Patterns & Design Principles
- Modular Design: Each major functionality (scraping, CLI handling, data serialization, evaluation) is decoupled into its own module or package. This separation facilitates maintainability and testing.
- Single Responsibility: Each scraper script is responsible for handling a specific data source, while the CLI commands orchestrate these actions.
- Consistent Data Format: The use of a standardized JSON schema (with designated objects like Task, Dataset, and SotaRow) provides a unified interface for all data outputs.
- Extensibility: New scrapers or evaluation scripts can be added without major refactoring, thanks to the clear module boundaries.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    subgraph "CLI Layer"
        CLI_Entry["CLI Entry Point"]:::cli
        CLI_Cmd["CLI Command Logic"]:::cli
    end

    CLI_Entry --> CLI_Cmd

    %% Scraper Engine Layer
    subgraph "Scraper Engine"
        Scrapers_General["General Scrapers"]:::scraper
        Scrapers_NLP["NLP-progress Scraper"]:::scraper
    end

    %% Data Handling & Serialization Layer
    subgraph "Data Handling & Serialization"
        Serialization["Serialization Logic"]:::data
        TaskDB["Task Database Models"]:::data
    end

    %% Data Storage Layer
    subgraph "Data Storage"
        Data_Dir["Data Directory & JSON Files"]:::data
    end

    %% Evaluation Module Layer
    subgraph "Evaluation Module"
        Eval_Cmd["Evaluation Command"]:::data
        Eval_Report["Evaluation Report File"]:::data
    end

    %% Utilities Layer
    subgraph "Utilities"
        Error["Error Handling Module"]:::util
        Scraper_Utils["Scraper Utility Functions"]:::util
    end

    %% Infrastructure & Testing Layer
    subgraph "Infrastructure & Testing"
        Build["Build & Setup Files"]:::infra
        Test["Test Suite"]:::test
    end

    %% Connections between layers
    CLI_Cmd -->|"triggers"| Scrapers_General
    CLI_Cmd -->|"triggers"| Scrapers_NLP

    Scrapers_General -->|"provides input"| Serialization
    Scrapers_NLP -->|"provides input"| Serialization

    Serialization -->|"ensures consistency"| TaskDB
    TaskDB -->|"stores data to"| Data_Dir

    Data_Dir -->|"feeds data to"| Eval_Cmd
    CLI_Cmd -->|"can also trigger"| Eval_Cmd
    Eval_Cmd -->|"generates"| Eval_Report

    %% Utilities support
    Error -.-> CLI_Cmd
    Error -.-> Scrapers_General
    Error -.-> Scrapers_NLP
    Scraper_Utils -.-> Scrapers_General
    Scraper_Utils -.-> Scrapers_NLP

    %% Styles
    classDef cli fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef scraper fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef data fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef util fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef infra fill:#e2e3e5,stroke:#383d41,stroke-width:2px;
    classDef test fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;

    %% Click Events
    click CLI_Entry "https://github.com/paperswithcode/sota-extractor/blob/master/sota_extractor/__main__.py"
    click CLI_Cmd "https://github.com/paperswithcode/sota-extractor/blob/master/sota_extractor/commands/cli.py"
    click Scrapers_General "https://github.com/paperswithcode/sota-extractor/tree/master/sota_extractor/scrapers"
    click Scrapers_NLP "https://github.com/paperswithcode/sota-extractor/tree/master/sota_extractor/scrapers/nlp_progress"
    click Serialization "https://github.com/paperswithcode/sota-extractor/blob/master/sota_extractor/serialization.py"
    click TaskDB "https://github.com/paperswithcode/sota-extractor/tree/master/sota_extractor/taskdb/v01"
    click Data_Dir "https://github.com/paperswithcode/sota-extractor/tree/master/data"
    click Eval_Cmd "https://github.com/paperswithcode/sota-extractor/blob/master/sota_extractor/commands/evaluate.py"
    click Eval_Report "https://github.com/paperswithcode/sota-extractor/blob/master/data/eval_all_report.csv"
    click Error "https://github.com/paperswithcode/sota-extractor/blob/master/sota_extractor/errors.py"
    click Scraper_Utils "https://github.com/paperswithcode/sota-extractor/blob/master/sota_extractor/scrapers/utils.py"
    click Test "https://github.com/paperswithcode/sota-extractor/tree/master/sota_extractor/tests"
```
