How internlm/internlm-math works
Step 1: Identify the Project Type and Purpose
• This is an open-source toolkit/repository for state-of-the-art bilingual math reasoning Large Language Models (LLMs).
• It bundles:
• Pretrained and fine-tuned InternLM2-Math models (solver, prover, verifier, augmentor)
• Datasets and proof benchmarks (Lean-Github, Lean-Workbook, MiniF2F)
• Inference “agent” code for tool-augmented reasoning and code interpretation
• Example scripts for HuggingFace and LMDeploy pipelines
• Evaluation scripts and metrics computation
Step 2: Analyze the File Structure
• Top-level: README.md, LICENSE, .gitignore
• agent/: Python code (math_agent.py) + requirements for tool-calling inference
• assets/: images and logos used in documentation
• examples/: lightweight inference demos (HuggingFace & LMDeploy)
• lean-github/: dataset compilation for Lean-GitHub proofs
• leanworkbook/: Lean-Workbook data, Lean header file, inference scripts (backtranslate, NLI, translate)
• minif2f/: MiniF2F dataset, proof-search Python scripts, eval scripts, shell wrappers, requirements
Step 3: Extract Insights from README
• Project layers:
1. Data Preparation (Lean-Github, Lean-Workbook, MiniF2F)
2. Model Checkpoints & Trainer (InternLM2-Math base & SFT)
3. Inference Agent (tool-calling via Python “agent”)
4. Evaluation & Metrics (compute_metrics.py, OpenCompass integration)
5. Deployment Examples (LMDeploy, HuggingFace transformers)
• Key technologies: Python, Lean 3/4, LLM frameworks (Turbomind, Transformers), JSONL datasets, shell scripting
Step 4: Main Components and Their Interactions
Data Layer
• Datasets in JSONL format (minif2f/data, lean-github source, leanworkbook/header.lean)
• Data-prep scripts (minif2f/proofsearch_*.py, lean-github README instructions)
Model Layer
• Pretrain & SFT checkpoints (InternLM2-Math-Base-7B/20B, InternLM2-Math-7B/20B)
• Training code lives outside this repo (referenced in InternLM core)
Inference Layer (agent)
• math_agent.py orchestrates prompt → model → tool (e.g., Python interpreter, Lean) → back to model
• Example pipelines: LMDeploy (TurbomindEngineConfig) and HuggingFace ChatTemplateConfig
Evaluation Layer
• compute_metrics.py for MiniF2F
• Shell scripts under minif2f/scripts for large-scale eval runs
• OpenCompass config for GSM8K/MATH benchmarks
External Services
• HuggingFace Hub (model & dataset storage)
• ModelScope, OpenXLab, LMDeploy for deployment
• ArXiv, paper links, technical reports
Step 5: Architectural Patterns and Design Principles
• Modular monorepo: clear separation of datasets, inference, examples, evaluation
• Pipeline pattern in “agent”: data in → preprocess → LLM call → tool call → postprocess → output
• Layered architecture: Data → Model → Inference Agent → Evaluation/Deployment
• Extensibility: add new datasets or inference tools by dropping modules into dedicated folders
• Core LLM engine surrounded by:
• Data modules feeding training/eval
• Inference agent orchestrating external tool calls
• Example scripts showing two deployment pathways (LMDeploy vs HF)
• That evaluation is pluggable (shell scripts, compute_metrics, OpenCompass)
Step 7: Diagramming Guidelines
• Datasets → Model (training/eval)
• Model ↔ Agent (inference requests & responses)
• Agent → External Tools (Python Interpreter, Lean) → Agent → User
• Evaluation scripts → Metrics → Reports
• Blue for Data modules
• Green for Model checkpoints
• Orange for Inference Agent and tool integration
• Purple for Examples/Deployment scripts
• Gray for External services (HuggingFace, ModelScope, OpenXLab)
Step 8: Call to Action for the Principal Software Engineer
• Drill down into each zone with sub-components (folders, scripts, datasets)
• Non-obvious integrations (e.g., how math_agent.py invokes Lean tactics or Python code)