# allenai/mathfish architecture

> Architecture diagram of the GitHub repository allenai/mathfish, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/allenai/mathfish
- Repository: https://github.com/allenai/mathfish

## Overview

##### Project Type and Purpose
- This is an open-source Python toolkit (“mathfish”) and CLI pipeline for transforming educational math problem datasets into language-model tasks, running models via API wrappers, and evaluating their performance on two task formats (verification and tagging). It is not a web or full-stack application, but a command-line/data-processing/ML-evaluation framework.

##### File-Structure Analysis
- Top-level directories:
- data/: raw and pre-generated problem datasets and few-shot CSVs
- mathfish/: core library package
– datasets/: loaders & dataset definitions (few_shot, random, tree-based, verification)
– preprocessors/: data augment, reformat, standardize pipelines
– modelWrappers/: API wrapper base, OpenAI dispatch, task-specific wrappers
– tree_retriever/: functions for retrieving curriculum-grounded solution trees
– evaluators/: verification and tagging evaluators, base classes
– utils.py: shared utilities
- scripts/: executable entry points for each pipeline stage
- tests/: unit tests for each component
- config & metadata: pyproject.toml, requirements.txt

##### Key Components
a. Data Layer
– Local CSV/JSONL files under data/
– Hugging Face datasets (mathfish, mathfish-tasks, achieve-the-core)
b. Preprocessing & Task Generation
– mathfish.preprocessors: data standardization, augmentation, reformatting
– scripts/output_dataset_verification.py
– scripts/output_dataset_tagging.py
– tree_traverser.py for self-guided tagging
c. Model Invocation
– mathfish.modelWrappers:
- base_wrapper.py (abstract retry/truncate logic)
- dispatch_openai_requests.py (API call handler)
- tagging_wrapper.py & verification_wrapper.py (prompt formatting & parsing)
– scripts/run_wrapper.py
d. Evaluation
– mathfish.evaluators: tagging_evaluator.py, verification_evaluator.py
– scripts/eval.py
e. External Services
– OpenAI API / other LLM endpoints
– Hugging Face dataset hub

##### Relationships & Data Flows
- Data Ingestion → Preprocessor → Task Generator → Wrapped Model Call → Model Output → Evaluator → Final Metrics
- tree_retriever is invoked within the tagging pipeline for constructing solution trees
- scripts orchestrate these steps in sequence or via CLI flags

##### Architectural Patterns & Principles
- Layered architecture: separation of concerns between data I/O, preprocessing, task creation, model interaction, evaluation
- Wrapper/adapter pattern for model API abstraction
- Modular design: each subpackage has a single responsibility, enabling extension (e.g., add a new modelWrapper)
- CLI orchestration for reproducibility and parameterization

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        FS_ENT["few_shot_entailment.csv"]:::data
        click FS_ENT "https://github.com/allenai/mathfish/blob/main/data/few_shot/few_shot_entailment.csv"
        FS_TAG["few_shot_tagging.csv"]:::data
        click FS_TAG "https://github.com/allenai/mathfish/blob/main/data/few_shot/few_shot_tagging.csv"
        GEN_IN["generated_problems_as_input_data.jsonl"]:::data
        click GEN_IN "https://github.com/allenai/mathfish/blob/main/data/generated_problems_as_input_data.jsonl"
        GEN_ANN["generated_problems_teacher_annotations.csv"]:::data
        click GEN_ANN "https://github.com/allenai/mathfish/blob/main/data/generated_problems_teacher_annotations.csv"
        GSM8K["gsm8k_test.jsonl"]:::data
        click GSM8K "https://github.com/allenai/mathfish/blob/main/data/gsm8k_test.jsonl"
        
        subgraph "Dataset Definitions"
            direction TB
            BASE_DS["base_dataset.py"]:::data
            click BASE_DS "https://github.com/allenai/mathfish/blob/main/mathfish/datasets/base_dataset.py"
            FS_TAG_DS["few_shot_tagging_exp.csv"]:::data
            click FS_TAG_DS "https://github.com/allenai/mathfish/blob/main/mathfish/datasets/few_shot_tagging_exp.csv"
            FS_VER_DS["few_shot_verification_exp.csv"]:::data
            click FS_VER_DS "https://github.com/allenai/mathfish/blob/main/mathfish/datasets/few_shot_verification_exp.csv"
            RAND_DS["random_tagger_dataset.py"]:::data
            click RAND_DS "https://github.com/allenai/mathfish/blob/main/mathfish/datasets/random_tagger_dataset.py"
            TREE_DS["tree_tagger_dataset.py"]:::data
            click TREE_DS "https://github.com/allenai/mathfish/blob/main/mathfish/datasets/tree_tagger_dataset.py"
            VER_DS["verification_dataset.py"]:::data
            click VER_DS "https://github.com/allenai/mathfish/blob/main/mathfish/datasets/verification_dataset.py"
        end
    end

    %% Preprocessing & Task Generation
    subgraph "Preprocessing & Task Generation" 
        direction TB
        DataStd["data_standardize.py"]:::preprocess
        click DataStd "https://github.com/allenai/mathfish/blob/main/mathfish/preprocessors/data_standardize.py"
        DataAug["data_augment.py"]:::preprocess
        click DataAug "https://github.com/allenai/mathfish/blob/main/mathfish/preprocessors/data_augment.py"
        DataRef["data_reformat.py"]:::preprocess
        click DataRef "https://github.com/allenai/mathfish/blob/main/mathfish/preprocessors/data_reformat.py"
        
        subgraph "Task Generation Scripts"
            direction TB
            ODV["output_dataset_verification.py"]:::script
            click ODV "https://github.com/allenai/mathfish/blob/main/scripts/output_dataset_verification.py"
            ODT["output_dataset_tagging.py"]:::script
            click ODT "https://github.com/allenai/mathfish/blob/main/scripts/output_dataset_tagging.py"
            TT["tree_traverser.py"]:::script
            click TT "https://github.com/allenai/mathfish/blob/main/scripts/tree_traverser.py"
        end
    end

    %% Tree Retriever
    subgraph "Tree Retriever"
        direction TB
        RETR["retrieve_tree.py"]:::preprocess
        click RETR "https://github.com/allenai/mathfish/blob/main/mathfish/tree_retriever/retrieve_tree.py"
        ATC["atc_map.py"]:::preprocess
        click ATC "https://github.com/allenai/mathfish/blob/main/mathfish/tree_retriever/atc_map.py"
    end

    %% Model Invocation
    subgraph "Model Invocation"
        direction TB
        BASE_WR["base_wrapper.py"]:::wrapper
        click BASE_WR "https://github.com/allenai/mathfish/blob/main/mathfish/modelWrappers/base_wrapper.py"
        DISP["dispatch_openai_requests.py"]:::wrapper
        click DISP "https://github.com/allenai/mathfish/blob/main/mathfish/modelWrappers/dispatch_openai_requests.py"
        TAG_WR["tagging_wrapper.py"]:::wrapper
        click TAG_WR "https://github.com/allenai/mathfish/blob/main/mathfish/modelWrappers/tagging_wrapper.py"
        VER_WR["verification_wrapper.py"]:::wrapper
        click VER_WR "https://github.com/allenai/mathfish/blob/main/mathfish/modelWrappers/verification_wrapper.py"
        RUN_WR["run_wrapper.py"]:::script
        click RUN_WR "https://github.com/allenai/mathfish/blob/main/scripts/run_wrapper.py"
    end

    %% External Services
    EXTERNAL["LLM API (OpenAI)"]:::external

    %% Evaluation
    subgraph "Evaluation"
        direction TB
        TAG_EVAL["tagging_evaluator.py"]:::eval
        click TAG_EVAL "https://github.com/allenai/mathfish/blob/main/mathfish/evaluators/tagging_evaluator.py"
        VER_EVAL["verification_evaluator.py"]:::eval
        click VER_EVAL "https://github.com/allenai/mathfish/blob/main/mathfish/evaluators/verification_evaluator.py"
        EVAL_SCRIPT["eval.py"]:::script
        click EVAL_SCRIPT "https://github.com/allenai/mathfish/blob/main/scripts/eval.py"
    end

    %% Configuration
    subgraph "Configuration"
        direction TB
        UTIL_ROOT["utils.py"]:::config
        click UTIL_ROOT "https://github.com/allenai/mathfish/blob/main/mathfish/utils.py"
        UTIL_SCRIPTS["utils.py"]:::config
        click UTIL_SCRIPTS "https://github.com/allenai/mathfish/blob/main/scripts/utils.py"
        PYPROJ["pyproject.toml"]:::config
        REQTXT["requirements.txt"]:::config
    end

    %% Tests
    subgraph "Tests"
        direction TB
        TEST_DS["tests for datasets"]:::test
        TEST_PP["tests for preprocessors"]:::test
        TEST_MW["tests for modelWrappers"]:::test
        TEST_EV["tests for evaluators"]:::test
    end

    %% Flow: Data -> Preprocess
    FS_ENT & FS_TAG & GEN_IN & GEN_ANN & GSM8K -->|CSV/JSONL| DataStd
    DataStd -->|cleaned data| DataAug
    DataAug -->|augmented data| DataRef

    %% Preprocessed to Task Generation
    DataRef -->|standard data| ODV
    DataRef -->|standard data| ODT

    %% Tagging flow with Tree retriever
    ODT -->|"task dataset"| TREE_DS
    ODT -->|"invoke tree retriever"| RETR
    RETR --> ATC
    ATC --> ODT

    %% Verification flow
    ODV --> VER_DS

    %% Scripts produce tasks to wrappers
    ODT & ODV -->|prompts| RUN_WR
    RUN_WR --> BASE_WR
    BASE_WR --> DISP
    DISP --> EXTERNAL
    EXTERNAL --> DISP
    DISP --> BASE_WR
    BASE_WR --> TAG_WR
    BASE_WR --> VER_WR

    %% Wrappers to Evaluators
    TAG_WR --> TAG_EVAL
    VER_WR --> VER_EVAL

    %% Evaluation to Metrics
    TAG_EVAL & VER_EVAL --> EVAL_SCRIPT
    EVAL_SCRIPT -->|"final metrics"| Output[/"Metrics Report"/]

    %% Config & Utils used by modules
    UTIL_ROOT & UTIL_SCRIPTS --> DataStd
    UTIL_ROOT & UTIL_SCRIPTS --> BASE_WR
    UTIL_ROOT & UTIL_SCRIPTS --> TAG_EVAL
    PYPROJ & REQTXT --> UTIL_ROOT

    %% Tests validate modules
    TEST_DS --> BASE_DS
    TEST_PP --> DataStd
    TEST_PP --> DataAug
    TEST_PP --> DataRef
    TEST_MW --> BASE_WR
    TEST_MW --> TAG_WR
    TEST_MW --> VER_WR
    TEST_EV --> TAG_EVAL
    TEST_EV --> VER_EVAL

    %% Styles
    classDef data fill:#a8d0e6,stroke:#333,stroke-width:1px,shape:cylinder
    classDef preprocess fill:#a8e6a8,stroke:#333,stroke-width:1px
    classDef wrapper fill:#fff59d,stroke:#333,stroke-width:1px
    classDef external fill:#ffcc80,stroke:#333,stroke-width:1px
    classDef eval fill:#ce93d8,stroke:#333,stroke-width:1px
    classDef script fill:#e0e0e0,stroke:#333,stroke-width:1px
    classDef test fill:#f28b82,stroke:#d32f2f,stroke-width:2px,stroke-dasharray:5 5
    classDef config fill:#f5f5f5,stroke:#999,stroke-width:1px
```
