How yifanzhang-pro/automathtext works
AutoMathText is a data-selection pipeline and curated mathematical-text dataset. Its three source-specific scripts load JSONL records, form prompts, score batches with Qwen-72B, and save scored, confidence-ranked JSONL output. The README describes the resulting dataset as drawing from web, arXiv, and code sources and being available through Hugging Face. The excerpts show script-level loading, model generation, and output handling; the README documents the published dataset, but the repository evidence shown does not establish a publishing step in these scripts.
Main entry points: process-web-data.py Web Selection · process-arxiv-data.py ArXiv Selection · process-code-data.py Code Selection
Components 7
Selection Pipelines
- Web Selection
src/process-web-data.py - ArXiv Selection
src/process-arxiv-data.py - Code Selection
src/process-code-data.py - Scored JSONL
Language Model
- Qwen-72B
Dataset Distribution
- AutoMathText Dataset
Other components
- Hugging Face Users
Connections 8
- Web Selection → Qwen-72B: generates prompts
src/process-web-data.py - Web Selection → Qwen-72B: requests scores
src/process-web-data.py - Web Selection → Scored JSONL: writes ranked batches
src/process-web-data.py - ArXiv Selection → Qwen-72B: requests scores
src/process-arxiv-data.py - ArXiv Selection → Scored JSONL: writes ranked batches
src/process-arxiv-data.py - Code Selection → Qwen-72B: requests scores
src/process-code-data.py - Code Selection → Scored JSONL: writes ranked batches
src/process-code-data.py - AutoMathText Dataset → Hugging Face Users: loaded via API
README.md