How paperswithcode/sota-extractor works
Main Components
• Command Line Interface (CLI) & Entry Points:
• The main CLI logic (located in sota_extractor/__main__.py and sota_extractor/commands/cli.py) is the entry point, handling user commands to run scrapers or evaluate data.
• Scraper Modules:
• A dedicated “scrapers” module (sota_extractor/scrapers) contains individual scraper scripts (e.g., chexpert.py, cityscapes.py, cmrc.py, etc.) for each external data source. Each scraper encapsulates logic to fetch and parse data from its respective website.
• Notice also a nested scraper for “nlp_progress”, which has its own substructure.
• Data Handling & Serialization:
• The “serialization.py” file manages the conversion of raw data into the structured JSON format described in the README.
• The “taskdb” sub-package (sota_extractor/taskdb/v01) provides models and schemas to represent the core entities (Task, Dataset, Sota, etc.) in a consistent way.
• Data Storage:
• The “data” folder holds both the static scraped outputs and the example JSON files (including tasks and metadata) that follow a defined structure.
• Evaluation Components:
• Commands like “evaluate.py” and references to the “eval_all_report.csv” in the README indicate an evaluation pipeline to check the performance of the extraction process.
• Utility & Error Handling:
• Utility modules (like “errors.py” and “utils.py”) are used to manage exceptions and common functions across the project.
Relationships & Interactions
• The CLI acts as a gateway to trigger the scrapers and evaluation routines.
• When a scraping command is invoked (e.g., python -m scrapers.squad), the corresponding scraper module initiates a request to the external service, processes the response and then serializes the result into the JSON format.
• The serialization logic interacts with the “taskdb” module to ensure data consistency (mapping tasks, datasets, and SOTA results).
• All resulting data is stored in the “data” directory.
• Evaluation commands pull the stored data (or freshly scraped data) and generate performance reports, such as the “eval_all_report.csv”.
Architectural Patterns & Design Principles
• Modular Design: Each major functionality (scraping, CLI handling, data serialization, evaluation) is decoupled into its own module or package. This separation facilitates maintainability and testing.
• Single Responsibility: Each scraper script is responsible for handling a specific data source, while the CLI commands orchestrate these actions.
• Consistent Data Format: The use of a standardized JSON schema (with designated objects like Task, Dataset, and SotaRow) provides a unified interface for all data outputs.
• Extensibility: New scrapers or evaluation scripts can be added without major refactoring, thanks to the clear module boundaries.