# cookieshop02/math-rl-agent architecture

> Architecture diagram of the GitHub repository cookieshop02/math-rl-agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cookieshop02/math-rl-agent
- Repository: https://github.com/cookieshop02/math-rl-agent

## Overview

1. **Overall shape**
   - This repo is split into two major subsystems: a **runtime math assistant** and an **offline RL training pipeline**.
   - The runtime app serves a browser UI and a Flask-style API; the training side prepares and evaluates a math-solving model fine-tuned with **GRPO**.

2. **User-facing application**
   - `static/index.html` is the only frontend asset in the tree, so the UI is a **single-page static client** rather than a multi-page web app.
   - The README indicates it supports two user flows: **math problem solving** and a **concept/Q&A assistant** side panel.

3. **Backend/API boundary**
   - `api.py` is the most likely HTTP server entrypoint in this tree, exposing the REST endpoints described in the README:
     - `POST /solve`
     - `POST /concept`
     - `GET /health`
   - `procfile` suggests the app is designed to run as a deployed web service on **Render**.

4. **Inference layer**
   - `setup.py` is the architecture’s inference bridge, despite the unusual filename.
   - Its job is to turn API requests into model calls and normalize the response into app-friendly fields like **solution**, **answer**, and **confidence**.

5. **Dual-model inference path**
   - The runtime is built around a **primary/fallback** design:
     - **Primary:** the fine-tuned Qwen2.5-Math model hosted remotely on a Kaggle GPU endpoint
     - **Fallback:** **Groq LLaMA-3.1** when the primary model endpoint is unavailable
   - This makes the service resilient to GPU downtime and avoids hard dependency on a single inference host.

6. **Training pipeline boundary**
   - `main.py` is the top-level entrypoint for training rather than serving.
   - The training flow is broken into separate modules, which keeps data loading, reward shaping, optimization, and evaluation isolated.

7. **Dataset and preprocessing**
   - `loader.py` is responsible for loading the **GSM8K** dataset used for training.
   - Based on the README, the training subset is relatively small, so this module likely prepares problem/answer pairs for the RL loop rather than doing large-scale ETL.

8. **Reward shaping**
   - `reward.py` defines the custom reward function used by GRPO.
   - The reward signal is explicitly multi-part: correctness of the final answer, presence of step-by-step reasoning, and penalties for missing or wrong answers.
   - This module is the key link between math task quality and policy optimization.

9. **Optimization loop**
   - `trainer.py` holds the GRPO training loop.
   - This is where multiple generations per question are scored, compared relative to each other, and used to update the policy while applying KL regularization to limit drift from the base model.

10. **Evaluation**
   - `eval.py` provides post-training evaluation of the fine-tuned model.
   - Architecturally, this is the validation boundary between the offline training system and the deployed inference service.

11. **Deployment/runtime dependencies**
   - `requirements.txt` captures the Python stack for both serving and training.
   - The README indicates important runtime environment variables:
     - `GROQ_API_KEY` for fallback inference
     - `HF_TOKEN` for HuggingFace access
   - These external services materially affect system behavior, since model availability depends on them.

12. **Data flow summary**
   - Browser submits a question → Flask API handles the request → `setup.py` routes to the primary model endpoint → if that fails, it falls back to Groq → response is returned to the UI.
   - Separately, offline training flows from `loader.py` → `trainer.py` + `reward.py` → `eval.py` → resulting model is hosted externally and reused by the live app.

13. **Important architectural boundary**
   - The live product is **not training the model in production**; training is completely separated from serving.
   - The production app is essentially a thin orchestration layer around externally hosted inference endpoints and a static UI.

Main entry points: [`trainer.py`](https://github.com/cookieshop02/math-rl-agent/blob/main/trainer.py) (GRPO loop), [`api.py`](https://github.com/cookieshop02/math-rl-agent/blob/main/api.py) (API server), [`setup.py`](https://github.com/cookieshop02/math-rl-agent/blob/main/setup.py) (Inference bridge), [`loader.py`](https://github.com/cookieshop02/math-rl-agent/blob/main/loader.py) (Dataset loader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime app"]
  node_ui["Browser UI<br/>static client<br/>[index.html]"]
  node_api["API server<br/>HTTP service<br/>[api.py]"]
  node_bridge["Inference bridge<br/>routing layer<br/>[setup.py]"]
  node_primary_model[("Qwen endpoint<br/>remote model")]
  node_fallback_model[("Groq fallback<br/>backup LLM")]
  node_deploy["Render deploy<br/>process spec"]
  node_hf(("HuggingFace<br/>external service"))
  node_groq(("Groq API<br/>external service"))
end

subgraph group_training["Training pipeline"]
  node_entry["Train entry<br/>training entrypoint<br/>[main.py]"]
  node_loader["Dataset loader<br/>data prep<br/>[loader.py]"]
  node_reward["Reward logic<br/>GRPO reward<br/>[reward.py]"]
  node_trainer["GRPO loop<br/>optimizer<br/>[trainer.py]"]
  node_eval["Evaluator<br/>validation<br/>[eval.py]"]
  node_gsm8k[("GSM8K data<br/>dataset")]
  node_model_artifact[("Fine-tuned model<br/>model artifact")]
end

node_ui -->|"user requests"| node_api
node_api -->|"dispatch"| node_bridge
node_bridge -->|"primary call"| node_primary_model
node_bridge -.->|"fallback call"| node_fallback_model
node_fallback_model -->|"via"| node_groq
node_primary_model -.->|"hosted via"| node_hf
node_deploy -->|"launches"| node_api
node_entry -->|"starts"| node_loader
node_loader -->|"reads"| node_gsm8k
node_entry -->|"runs"| node_trainer
node_trainer -->|"scores"| node_reward
node_trainer -->|"consumes"| node_loader
node_trainer -->|"produces"| node_model_artifact
node_eval -->|"checks"| node_model_artifact
node_model_artifact -->|"deployed as"| node_primary_model
node_api -->|"responses"| node_ui

click node_ui "https://github.com/cookieshop02/math-rl-agent/blob/main/static/index.html"
click node_api "https://github.com/cookieshop02/math-rl-agent/blob/main/api.py"
click node_bridge "https://github.com/cookieshop02/math-rl-agent/blob/main/setup.py"
click node_deploy "https://github.com/cookieshop02/math-rl-agent/tree/main/procfile"
click node_entry "https://github.com/cookieshop02/math-rl-agent/blob/main/main.py"
click node_loader "https://github.com/cookieshop02/math-rl-agent/blob/main/loader.py"
click node_reward "https://github.com/cookieshop02/math-rl-agent/blob/main/reward.py"
click node_trainer "https://github.com/cookieshop02/math-rl-agent/blob/main/trainer.py"
click node_eval "https://github.com/cookieshop02/math-rl-agent/blob/main/eval.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ui,node_api,node_bridge,node_primary_model,node_fallback_model,node_deploy,node_hf,node_groq toneBlue
class node_entry,node_loader,node_reward,node_trainer,node_eval,node_gsm8k,node_model_artifact toneAmber
```

## Components

### Runtime app

Static UI, HTTP API, and inference routing for live math assistance.

- [Browser UI](https://github.com/cookieshop02/math-rl-agent/blob/main/static/index.html): static client · Single-page front end for solving math problems and asking concept questions. · `static/index.html`
- [API server](https://github.com/cookieshop02/math-rl-agent/blob/main/api.py): HTTP service · Flask-style entrypoint exposing solve, concept, and health endpoints. · `api.py`
- [Inference bridge](https://github.com/cookieshop02/math-rl-agent/blob/main/setup.py): routing layer · Turns API requests into model calls and normalizes solution, answer, and confidence output. · `setup.py`
- **Qwen endpoint**: remote model · Primary hosted math model used for live inference.
- **Groq fallback**: backup LLM · Fallback inference path when the primary model endpoint is unavailable.
- [Render deploy](https://github.com/cookieshop02/math-rl-agent/tree/main/procfile): process spec · Procfile-based service launch configuration for deployment. · `procfile`
- **HuggingFace**: external service · External model access and token-backed dependency for training or hosting.
- **Groq API**: external service · Fallback LLM provider used when the primary hosted model is unavailable.

### Training pipeline

Offline GRPO training and evaluation for the math model.

- [Train entry](https://github.com/cookieshop02/math-rl-agent/blob/main/main.py): training entrypoint · Top-level script that starts the offline training workflow. · `main.py`
- [Dataset loader](https://github.com/cookieshop02/math-rl-agent/blob/main/loader.py): data prep · Loads and prepares GSM8K problem/answer pairs for training. · `loader.py`
- [Reward logic](https://github.com/cookieshop02/math-rl-agent/blob/main/reward.py): GRPO reward · Computes correctness, reasoning, and penalty signals for policy optimization. · `reward.py`
- [GRPO loop](https://github.com/cookieshop02/math-rl-agent/blob/main/trainer.py): optimizer · Runs multi-sample training, scoring, comparison, and policy updates. · `trainer.py`
- [Evaluator](https://github.com/cookieshop02/math-rl-agent/blob/main/eval.py): validation · Checks the fine-tuned model after training and gates model quality. · `eval.py`
- **GSM8K data**: dataset · Math word-problem dataset used to train and evaluate the model.
- **Fine-tuned model**: model artifact · Result of GRPO training that is hosted externally for serving.

## Connections

- Browser UI → API server: user requests · Browser submits solve and concept requests to the HTTP backend.
- API server → Inference bridge: dispatch · API layer forwards requests to the inference bridge.
- Inference bridge → Qwen endpoint: primary call · Sends inference to the hosted fine-tuned math model.
- Inference bridge → Groq fallback: fallback call · Uses Groq when the primary model endpoint fails.
- Groq fallback → Groq API: via · Fallback inference is served by Groq.
- Qwen endpoint → HuggingFace: hosted via · Primary model availability depends on external model hosting and access.
- Render deploy → API server: launches · Deployment process starts the API service.
- Train entry → Dataset loader: starts · Training entrypoint initializes dataset loading.
- Dataset loader → GSM8K data: reads · Loads the GSM8K training data.
- Train entry → GRPO loop: runs · Training entrypoint launches the GRPO optimization loop.
- GRPO loop → Reward logic: scores · Trainer uses reward shaping to evaluate sampled outputs.
- GRPO loop → Dataset loader: consumes · Training loop consumes prepared examples from the loader.
- GRPO loop → Fine-tuned model: produces · Optimization yields the fine-tuned model artifact.
- Evaluator → Fine-tuned model: checks · Evaluator validates the trained model artifact.
- Fine-tuned model → Qwen endpoint: deployed as · Validated model is hosted for live inference.
- API server → Browser UI: responses · API returns solution and confidence data to the browser UI.
