# eth-sri/matharena architecture

> Architecture diagram of the GitHub repository eth-sri/matharena, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/eth-sri/matharena
- Repository: https://github.com/eth-sri/matharena

## Overview

##### Project Type and Purpose
- MathArena is primarily an evaluation platform that runs LLMs on math competitions (e.g., AIMEs, USAMO, CMIMC, HMMT). Its purpose is to configure models, run them on LaTeX-based problem sets, grade their answers (with optional human judgments), and display the results using a web interface.
- The README confirms that it supports evaluating new models, adding competitions, running grading pipelines, and viewing results through a local webserver.

##### Analyzing the File Structure
- Notice the main top-level directories:
- “configs”: Contains configuration YAML files for various API providers and models (e.g., OpenAI, Anthropic, DeepSeek, Gemini, Qwen, etc.). These files specify model names, API keys, and parameters.
- “data”: Houses competitions. Each competition (AIME, HMNT, USAMO, etc.) is represented as its own subdirectory with its problems (as LaTeX files), configuration files, answers, and sometimes grading schemes.
- “src/matharena”: Contains the core business logic – from APIs to configuration management, runners, grading, parsing, and miscellaneous utilities.
- “scripts”: Provides executable scripts for running the evaluation pipeline (run.py, run_multiple.py), checking LaTeX, the judging pipeline (answers_to_latex.py, merge_judgments.py), grading, and launching the web server (app.py).
- “latex”: Used to compile merged PDFs from problems and model answers (for verification or judging).
- Other files (like the LICENSE, .gitignore, README_judges.md) support project management and documentation.

##### Identifying Main Components
- Model Configuration Component: Uses “configs” to set parameters for different models and API providers, defining how each model is queried (including parameters like temperature, max_tokens, etc.).
- Competition Data Component: “data” stores problem statements, competition-specific instructions, answers, and grading schemes.
- Core Evaluation Engine: Found in “src/matharena”, which includes modules for API interactions (api.py), executing models (runner.py), parsing results (parser.py and parse_manual.py), and grading (grader.py, possible_issues.py).
- Utility and Pipeline Scripts: “scripts” orchestrates the overall pipeline: running models against competitions (run.py, run_multiple.py), checking LaTeX outputs, merging results, and handling judgments for human grading.
- Web/App Component: “scripts/app.py” starts a local server to display and review results.
- Document Generation Component: “latex” holds merged LaTeX files that compile into PDFs for both competition problem compilations and graded outputs.

##### Relationships and Data Flows
- Competition problems (in data) pass into runner scripts that reference configuration files (from configs). The “src/matharena” modules drive this process.
- Model outputs are generated and processed – some are directly parsed and scored, while others may require manual parsing (via parse_manual.py).
- Grading can be automated (via scripts/grade.py) or use the external judging pipeline (using scripts/judge/*).
- Final results (and optionally grading PDFs) are published via the web server (scripts/app.py) for user inspection.

##### Architectural Patterns and Design Principles
- Separation of Concerns: Notice the clear segmentation between configuration management (configs), competition data (data), core evaluation logic (src), and runtime pipelines (scripts).
- Extensibility: The modular nature (adding new competitions, configurations or even a new grading mode) shows an emphasis on being easily extendable.
- Pipeline Orchestration: There’s a sequential and concurrent processing of problems (with special retry and error-handling logic in the scripts), indicating an orchestration that handles queueing and parallelism.
- Microservices-like Components (internally): Although not deployed as separate network services, the system’s separation into distinct modules (evaluation, grading, PDF generation, and web-based result viewing) suggests a micro-component approach.

##### Suggested Elements
- Clear labels for each key component:
– Model Configurations (“configs”)
– Competition Data (“data”)
– Core Engine (“src/matharena” modules)
– Pipeline Scripts (“scripts” – run, grade, judge utilities)
– Document Generation (“latex” PDFs)
– Web Interface (“scripts/app.py” for results viewing)
– Show the flow from configuration and competition data input to evaluation.
– Indicate data processing flows (e.g., “Run Model” → “Generate LaTeX output” → “Compile PDF” → “Display Results”).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Input Layer"
        MC["Model Configuration Manager"]
        CD["Competition Data Repository"]
    end

    subgraph "Processing Layer"
        CEE["Core Evaluation Engine"]
        PS["Pipeline Scripts"]
    end

    subgraph "Output Layer"
        WI["Web Interface / Result Viewer"]
        DG["Document Generation Component"]
    end

    MC -->|"load_config"| CEE
    CD -->|"provide_data"| CEE
    CEE -->|"run_evaluation"| PS
    PS -->|"display_results"| WI
    PS -->|"generate_documents"| DG

    classDef config fill:#f1c40f,stroke:#333,stroke-width:2px;
    classDef repos fill:#e67e22,stroke:#333,stroke-width:2px;
    classDef engine fill:#3498db,stroke:#333,stroke-width:2px;
    classDef scripts fill:#2ecc71,stroke:#333,stroke-width:2px;
    classDef web fill:#9b59b6,stroke:#333,stroke-width:2px;
    classDef doc fill:#e74c3c,stroke:#333,stroke-width:2px;

    class MC config;
    class CD repos;
    class CEE engine;
    class PS scripts;
    class WI web;
    class DG doc;

    click MC "https://github.com/eth-sri/matharena/tree/main/configs/"
    click CD "https://github.com/eth-sri/matharena/tree/main/data/"
    click CEE "https://github.com/eth-sri/matharena/tree/main/src/matharena/"
    click PS "https://github.com/eth-sri/matharena/tree/main/scripts/"
    click WI "https://github.com/eth-sri/matharena/blob/main/scripts/app.py"
    click DG "https://github.com/eth-sri/matharena/tree/main/latex/"
```
