# yifanzhang-pro/automathtext architecture

> Architecture diagram of the GitHub repository yifanzhang-pro/automathtext, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yifanzhang-pro/automathtext
- Repository: https://github.com/yifanzhang-pro/automathtext

## Overview

AutoMathText is a data-selection pipeline and curated mathematical-text dataset. Its three source-specific scripts load JSONL records, form prompts, score batches with Qwen-72B, and save scored, confidence-ranked JSONL output. The README describes the resulting dataset as drawing from web, arXiv, and code sources and being available through Hugging Face. The excerpts show script-level loading, model generation, and output handling; the README documents the published dataset, but the repository evidence shown does not establish a publishing step in these scripts.

Main entry points: [`src/process-web-data.py`](https://github.com/yifanzhang-pro/automathtext/blob/master/src/process-web-data.py) (Web Selection), [`src/process-arxiv-data.py`](https://github.com/yifanzhang-pro/automathtext/blob/master/src/process-arxiv-data.py) (ArXiv Selection), [`src/process-code-data.py`](https://github.com/yifanzhang-pro/automathtext/blob/master/src/process-code-data.py) (Code Selection)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_pipelines["Selection Pipelines"]
  node_web["Web Selection"]
  node_arxiv["ArXiv Selection"]
  node_code["Code Selection"]
  node_scored[("Scored JSONL")]
end

subgraph group_model["Language Model"]
  node_qwen["Qwen-72B"]
end

subgraph group_distribution["Dataset Distribution"]
  node_automathtext[("AutoMathText Dataset")]
end

node_huggingface(("Hugging Face Users"))

node_web -->|"generates prompts"| node_qwen
node_web -->|"requests scores"| node_qwen
node_web -->|"writes ranked batches"| node_scored
node_arxiv -->|"requests scores"| node_qwen
node_arxiv -->|"writes ranked batches"| node_scored
node_code -->|"requests scores"| node_qwen
node_code -->|"writes ranked batches"| node_scored
node_automathtext -->|"loaded via API"| node_huggingface

click node_web "https://github.com/yifanzhang-pro/automathtext/blob/master/src/process-web-data.py"
click node_arxiv "https://github.com/yifanzhang-pro/automathtext/blob/master/src/process-arxiv-data.py"
click node_code "https://github.com/yifanzhang-pro/automathtext/blob/master/src/process-code-data.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_web,node_arxiv,node_code,node_scored,node_huggingface toneBlue
class node_qwen toneAmber
class node_automathtext toneMint
```

## Components

### Selection Pipelines

- [Web Selection](https://github.com/yifanzhang-pro/automathtext/blob/master/src/process-web-data.py): `src/process-web-data.py`
- [ArXiv Selection](https://github.com/yifanzhang-pro/automathtext/blob/master/src/process-arxiv-data.py): `src/process-arxiv-data.py`
- [Code Selection](https://github.com/yifanzhang-pro/automathtext/blob/master/src/process-code-data.py): `src/process-code-data.py`
- **Scored JSONL**

### Language Model

- **Qwen-72B**

### Dataset Distribution

- **AutoMathText Dataset**

### Other components

- **Hugging Face Users**

## Connections

- Web Selection → Qwen-72B: generates prompts (evidence: `src/process-web-data.py`)
- Web Selection → Qwen-72B: requests scores (evidence: `src/process-web-data.py`)
- Web Selection → Scored JSONL: writes ranked batches (evidence: `src/process-web-data.py`)
- ArXiv Selection → Qwen-72B: requests scores (evidence: `src/process-arxiv-data.py`)
- ArXiv Selection → Scored JSONL: writes ranked batches (evidence: `src/process-arxiv-data.py`)
- Code Selection → Qwen-72B: requests scores (evidence: `src/process-code-data.py`)
- Code Selection → Scored JSONL: writes ranked batches (evidence: `src/process-code-data.py`)
- AutoMathText Dataset → Hugging Face Users: loaded via API (evidence: `README.md`)