How eth-sri/matharena works
Project Type and Purpose
• MathArena is primarily an evaluation platform that runs LLMs on math competitions (e.g., AIMEs, USAMO, CMIMC, HMMT). Its purpose is to configure models, run them on LaTeX-based problem sets, grade their answers (with optional human judgments), and display the results using a web interface.
• The README confirms that it supports evaluating new models, adding competitions, running grading pipelines, and viewing results through a local webserver.
Analyzing the File Structure
• Notice the main top-level directories:
• “configs”: Contains configuration YAML files for various API providers and models (e.g., OpenAI, Anthropic, DeepSeek, Gemini, Qwen, etc.). These files specify model names, API keys, and parameters.
• “data”: Houses competitions. Each competition (AIME, HMNT, USAMO, etc.) is represented as its own subdirectory with its problems (as LaTeX files), configuration files, answers, and sometimes grading schemes.
• “src/matharena”: Contains the core business logic – from APIs to configuration management, runners, grading, parsing, and miscellaneous utilities.
• “scripts”: Provides executable scripts for running the evaluation pipeline (run.py, run_multiple.py), checking LaTeX, the judging pipeline (answers_to_latex.py, merge_judgments.py), grading, and launching the web server (app.py).
• “latex”: Used to compile merged PDFs from problems and model answers (for verification or judging).
• Other files (like the LICENSE, .gitignore, README_judges.md) support project management and documentation.
Identifying Main Components
• Model Configuration Component: Uses “configs” to set parameters for different models and API providers, defining how each model is queried (including parameters like temperature, max_tokens, etc.).
• Competition Data Component: “data” stores problem statements, competition-specific instructions, answers, and grading schemes.
• Core Evaluation Engine: Found in “src/matharena”, which includes modules for API interactions (api.py), executing models (runner.py), parsing results (parser.py and parse_manual.py), and grading (grader.py, possible_issues.py).
• Utility and Pipeline Scripts: “scripts” orchestrates the overall pipeline: running models against competitions (run.py, run_multiple.py), checking LaTeX outputs, merging results, and handling judgments for human grading.
• Web/App Component: “scripts/app.py” starts a local server to display and review results.
• Document Generation Component: “latex” holds merged LaTeX files that compile into PDFs for both competition problem compilations and graded outputs.
Relationships and Data Flows
• Competition problems (in data) pass into runner scripts that reference configuration files (from configs). The “src/matharena” modules drive this process.
• Model outputs are generated and processed – some are directly parsed and scored, while others may require manual parsing (via parse_manual.py).
• Grading can be automated (via scripts/grade.py) or use the external judging pipeline (using scripts/judge/*).
• Final results (and optionally grading PDFs) are published via the web server (scripts/app.py) for user inspection.
Architectural Patterns and Design Principles
• Separation of Concerns: Notice the clear segmentation between configuration management (configs), competition data (data), core evaluation logic (src), and runtime pipelines (scripts).
• Extensibility: The modular nature (adding new competitions, configurations or even a new grading mode) shows an emphasis on being easily extendable.
• Pipeline Orchestration: There’s a sequential and concurrent processing of problems (with special retry and error-handling logic in the scripts), indicating an orchestration that handles queueing and parallelism.
• Microservices-like Components (internally): Although not deployed as separate network services, the system’s separation into distinct modules (evaluation, grading, PDF generation, and web-based result viewing) suggests a micro-component approach.
Suggested Elements
• Clear labels for each key component:
– Model Configurations (“configs”)
– Competition Data (“data”)
– Core Engine (“src/matharena” modules)
– Pipeline Scripts (“scripts” – run, grade, judge utilities)
– Document Generation (“latex” PDFs)
– Web Interface (“scripts/app.py” for results viewing)
– Show the flow from configuration and competition data input to evaluation.
– Indicate data processing flows (e.g., “Run Model” → “Generate LaTeX output” → “Compile PDF” → “Display Results”).