How bobdavis84/autoresearch works
1. Repository purpose
- This repo is a small, single-GPU LLM research sandbox for autonomous experimentation.
- The core workflow is: change model code, run a short training job, compare validation quality, and keep or discard the change.
2. Primary architectural boundary
- The codebase is intentionally split into a fixed infrastructure layer and an editable research layer.
- prepare.py is treated as stable support code.
- train.py is the main mutable experiment surface.
- program.md is the human-authored instruction layer that steers the agent.
3. Data and preprocessing subsystem
- prepare.py owns one-time setup: dataset download, tokenizer training, and fixed constants.
- It also provides runtime utilities used by training, especially dataloading and evaluation support.
- This file is the bridge from raw text data to model-ready token sequences.
4. Training subsystem
- train.py contains the full model and optimization stack: GPT model definition, Muon + AdamW optimizer logic, and the training loop.
- This is where architecture changes happen: model depth, attention pattern, batch size, hyperparameters, or optimizer details.
- The file is designed to be edited repeatedly by the autonomous agent.
5. Agent orchestration layer
- program.md acts as the “control program” for the research agent.
- It describes how the agent should behave, what to edit, and how to iterate.
- The human updates this file to change the research strategy without touching training code directly.
6. Experiment loop and feedback flow
- The loop is: agent reads program.md, edits train.py, training runs for a fixed wall-clock budget, then validation results determine whether the change is promising.
- The feedback signal is val_bpb (validation bits per byte), where lower is better.
- This makes every run comparable within the same time budget.
7. Compute/runtime assumptions
- The repo is optimized for a single NVIDIA GPU and a 5-minute training budget per run.
- That fixed runtime constraint is part of the architecture: it shapes how model size, batch size, and evaluation cost are chosen.
- The runtime is intentionally simple rather than distributed or multi-node.
8. Technology stack
- The project is Python-based and managed with uv.
- PyTorch is the core ML runtime.
- The dependency and environment state live in pyproject.toml and uv.lock.
9. Supporting files and tooling
- setup.sh likely automates environment/bootstrap steps.
- README.linux.md suggests platform-specific setup guidance.
- analysis.ipynb is for offline exploration or inspection, not part of the main execution path.
10. Repository intent from an architecture perspective
- The design minimizes moving parts: one stable data/prep layer, one editable training file, one agent instruction file.
- That separation makes it easy to diagram a closed-loop research system: data preparation feeds training, training produces metrics, and the agent uses those metrics to decide the next code change.
Main entry points: train.py Training · analysis.ipynb Analysis · prepare.py Prepare · program.md Program
Components 13
Fixed infra
Stable setup, data preparation, and runtime support code.
- Prepare · Fixed infrastructure for dataset setup, tokenizer training, constants, and shared runtime utilities.
prepare.py - Data setup · One-time dataset download and tokenizer training.
prepare.py - Dataloader · Builds model-ready token batches from prepared text data.
prepare.py - Eval support · Shared evaluation helpers used to measure validation quality consistently.
prepare.py
Research loop
Editable experiment surface and agent control path.
- Program · Human-authored instructions that steer the research agent's edits and iteration strategy.
program.md - Training · Main mutable experiment surface containing the model, optimizer, and training loop.
train.py - GPT model · Transformer/GPT architecture definition and related forward logic.
train.py - Optimizers · Muon and AdamW optimization logic and parameter handling.
train.py - Train loop · Runs short training jobs under a fixed budget and evaluates changes.
train.py - val_bpb · Validation bits-per-byte feedback signal used to compare runs; lower is better.
Connections 14
- Program → Training: directs edits · Instruction layer tells the agent how to mutate the training code.
- Prepare → Training: supplies utils · Training imports fixed preprocessing, dataloading, and evaluation helpers.
- Data setup → Dataloader: feeds batches · Prepared data and tokenizer outputs are turned into trainable sequences.
- Dataloader → Training: inputs · Batches flow into the mutable training code.
- Eval support → val_bpb: computes · Evaluation helpers produce the validation metric used for comparison.
- Training → GPT model: contains · The training file defines the core model architecture.
- Training → Optimizers: contains · The training file defines optimizer configuration and update logic.
- Training → Train loop: runs · The main execution path is the short-budget training loop.
- Train loop → val_bpb: reports · Each run ends with a validation score for the agent to judge.
- val_bpb → Program: guides next step · The metric closes the feedback loop that informs the next edit.
- Env → Training: supports · Python environment and dependencies enable training execution.
- Bootstrap → Env: bootstraps · Setup script prepares the managed environment.
- Analysis → Training: inspects · Notebook is used for offline analysis of the experiment code and results.
- Analysis → Prepare: inspects · Notebook may also examine preprocessing and evaluation behavior.