How allenai/mathfish works
Project Type and Purpose
• This is an open-source Python toolkit (“mathfish”) and CLI pipeline for transforming educational math problem datasets into language-model tasks, running models via API wrappers, and evaluating their performance on two task formats (verification and tagging). It is not a web or full-stack application, but a command-line/data-processing/ML-evaluation framework.
File-Structure Analysis
• Top-level directories:
• data/: raw and pre-generated problem datasets and few-shot CSVs
• mathfish/: core library package
– datasets/: loaders & dataset definitions (few_shot, random, tree-based, verification)
– preprocessors/: data augment, reformat, standardize pipelines
– modelWrappers/: API wrapper base, OpenAI dispatch, task-specific wrappers
– tree_retriever/: functions for retrieving curriculum-grounded solution trees
– evaluators/: verification and tagging evaluators, base classes
– utils.py: shared utilities
• scripts/: executable entry points for each pipeline stage
• tests/: unit tests for each component
• config & metadata: pyproject.toml, requirements.txt
Key Components
a. Data Layer
– Local CSV/JSONL files under data/
– Hugging Face datasets (mathfish, mathfish-tasks, achieve-the-core)
b. Preprocessing & Task Generation
– mathfish.preprocessors: data standardization, augmentation, reformatting
– scripts/output_dataset_verification.py
– scripts/output_dataset_tagging.py
– tree_traverser.py for self-guided tagging
c. Model Invocation
– mathfish.modelWrappers:
• base_wrapper.py (abstract retry/truncate logic)
• dispatch_openai_requests.py (API call handler)
• tagging_wrapper.py & verification_wrapper.py (prompt formatting & parsing)
– scripts/run_wrapper.py
d. Evaluation
– mathfish.evaluators: tagging_evaluator.py, verification_evaluator.py
– scripts/eval.py
e. External Services
– OpenAI API / other LLM endpoints
– Hugging Face dataset hub
Relationships & Data Flows
• Data Ingestion → Preprocessor → Task Generator → Wrapped Model Call → Model Output → Evaluator → Final Metrics
• tree_retriever is invoked within the tagging pipeline for constructing solution trees
• scripts orchestrate these steps in sequence or via CLI flags
Architectural Patterns & Principles
• Layered architecture: separation of concerns between data I/O, preprocessing, task creation, model interaction, evaluation
• Wrapper/adapter pattern for model API abstraction
• Modular design: each subpackage has a single responsibility, enabling extension (e.g., add a new modelWrapper)
• CLI orchestration for reproducibility and parameterization