How cookieshop02/math-rl-agent works
1. Overall shape
- This repo is split into two major subsystems: a runtime math assistant and an offline RL training pipeline.
- The runtime app serves a browser UI and a Flask-style API; the training side prepares and evaluates a math-solving model fine-tuned with GRPO.
2. User-facing application
- static/index.html is the only frontend asset in the tree, so the UI is a single-page static client rather than a multi-page web app.
- The README indicates it supports two user flows: math problem solving and a concept/Q&A assistant side panel.
3. Backend/API boundary
- api.py is the most likely HTTP server entrypoint in this tree, exposing the REST endpoints described in the README:
- POST /solve
- POST /concept
- GET /health
- procfile suggests the app is designed to run as a deployed web service on Render.
4. Inference layer
- setup.py is the architecture’s inference bridge, despite the unusual filename.
- Its job is to turn API requests into model calls and normalize the response into app-friendly fields like solution, answer, and confidence.
5. Dual-model inference path
- The runtime is built around a primary/fallback design:
- Primary: the fine-tuned Qwen2.5-Math model hosted remotely on a Kaggle GPU endpoint
- Fallback: Groq LLaMA-3.1 when the primary model endpoint is unavailable
- This makes the service resilient to GPU downtime and avoids hard dependency on a single inference host.
6. Training pipeline boundary
- main.py is the top-level entrypoint for training rather than serving.
- The training flow is broken into separate modules, which keeps data loading, reward shaping, optimization, and evaluation isolated.
7. Dataset and preprocessing
- loader.py is responsible for loading the GSM8K dataset used for training.
- Based on the README, the training subset is relatively small, so this module likely prepares problem/answer pairs for the RL loop rather than doing large-scale ETL.
8. Reward shaping
- reward.py defines the custom reward function used by GRPO.
- The reward signal is explicitly multi-part: correctness of the final answer, presence of step-by-step reasoning, and penalties for missing or wrong answers.
- This module is the key link between math task quality and policy optimization.
9. Optimization loop
- trainer.py holds the GRPO training loop.
- This is where multiple generations per question are scored, compared relative to each other, and used to update the policy while applying KL regularization to limit drift from the base model.
10. Evaluation
- eval.py provides post-training evaluation of the fine-tuned model.
- Architecturally, this is the validation boundary between the offline training system and the deployed inference service.
11. Deployment/runtime dependencies
- requirements.txt captures the Python stack for both serving and training.
- The README indicates important runtime environment variables:
- GROQ_API_KEY for fallback inference
- HF_TOKEN for HuggingFace access
- These external services materially affect system behavior, since model availability depends on them.
12. Data flow summary
- Browser submits a question → Flask API handles the request → setup.py routes to the primary model endpoint → if that fails, it falls back to Groq → response is returned to the UI.
- Separately, offline training flows from loader.py → trainer.py + reward.py → eval.py → resulting model is hosted externally and reused by the live app.
13. Important architectural boundary
- The live product is not training the model in production; training is completely separated from serving.
- The production app is essentially a thin orchestration layer around externally hosted inference endpoints and a static UI.
Main entry points: trainer.py GRPO loop · api.py API server · setup.py Inference bridge · loader.py Dataset loader
Components 15
Runtime app
Static UI, HTTP API, and inference routing for live math assistance.
- Browser UI · Single-page front end for solving math problems and asking concept questions.
static/index.html - API server · Flask-style entrypoint exposing solve, concept, and health endpoints.
api.py - Inference bridge · Turns API requests into model calls and normalizes solution, answer, and confidence output.
setup.py - Qwen endpoint · Primary hosted math model used for live inference.
- Groq fallback · Fallback inference path when the primary model endpoint is unavailable.
- Render deploy · Procfile-based service launch configuration for deployment.
procfile - HuggingFace · External model access and token-backed dependency for training or hosting.
- Groq API · Fallback LLM provider used when the primary hosted model is unavailable.
Training pipeline
Offline GRPO training and evaluation for the math model.
- Train entry · Top-level script that starts the offline training workflow.
main.py - Dataset loader · Loads and prepares GSM8K problem/answer pairs for training.
loader.py - Reward logic · Computes correctness, reasoning, and penalty signals for policy optimization.
reward.py - GRPO loop · Runs multi-sample training, scoring, comparison, and policy updates.
trainer.py - Evaluator · Checks the fine-tuned model after training and gates model quality.
eval.py - GSM8K data · Math word-problem dataset used to train and evaluate the model.
- Fine-tuned model · Result of GRPO training that is hosted externally for serving.
Connections 16
- Browser UI → API server: user requests · Browser submits solve and concept requests to the HTTP backend.
- API server → Inference bridge: dispatch · API layer forwards requests to the inference bridge.
- Inference bridge → Qwen endpoint: primary call · Sends inference to the hosted fine-tuned math model.
- Inference bridge → Groq fallback: fallback call · Uses Groq when the primary model endpoint fails.
- Groq fallback → Groq API: via · Fallback inference is served by Groq.
- Qwen endpoint → HuggingFace: hosted via · Primary model availability depends on external model hosting and access.
- Render deploy → API server: launches · Deployment process starts the API service.
- Train entry → Dataset loader: starts · Training entrypoint initializes dataset loading.
- Dataset loader → GSM8K data: reads · Loads the GSM8K training data.
- Train entry → GRPO loop: runs · Training entrypoint launches the GRPO optimization loop.
- GRPO loop → Reward logic: scores · Trainer uses reward shaping to evaluate sampled outputs.
- GRPO loop → Dataset loader: consumes · Training loop consumes prepared examples from the loader.
- GRPO loop → Fine-tuned model: produces · Optimization yields the fine-tuned model artifact.
- Evaluator → Fine-tuned model: checks · Evaluator validates the trained model artifact.
- Fine-tuned model → Qwen endpoint: deployed as · Validated model is hosted for live inference.
- API server → Browser UI: responses · API returns solution and confidence data to the browser UI.