How thesaifhub/adaptive-compute-transformer works
Adaptive-Compute Transformer is a research system for training token-routing decoder transformers and testing whether reduced compute preserves language-model quality and improves real inference latency. The main workflow prepares WikiText-103 data, trains baseline or adaptive models, then evaluates adaptive checkpoints with hard-skip inference and benchmarks their latency against a dense baseline. Analysis scripts compare quality and speed across router budgets and visualize token routing. The excerpts show key model, training, hard-skip, and benchmark behavior; links for unsampled data and analysis modules follow the README and repository tree.
Main entry points: experiments Run checkpoints · train_adaptive.py Adaptive training · eval_hard_skip.py Hard-skip evaluation · benchmark_speed.py Latency benchmark
Components 21
Data and configuration
- Model configs
src/config.py - Dataset and batches
src/data.py
Transformer and routing
- Baseline transformer
src/model.py - Adaptive transformer
src/model.py - Router designs
src/router.py - Training losses
src/losses.py
Training and evaluation
- Baseline training
src/train.py - Adaptive training
src/train_adaptive.py - Run checkpoints
experiments - Hard-skip inference
src/hard_skip.py - Hard-skip evaluation
src/eval_hard_skip.py
Measurement and analysis
- Latency benchmark
src/benchmark_speed.py - Capacity diagnostics
src/diagnose_offcapacity.py - Routing visualization
src/visualize_routing.py - Adaptive quality curve
src/plot_curve.py - Top-k quality curve
src/plot_topk_curve.py - Quality-speed chart
src/plot_speed_vs_accuracy.py - Evaluation results
results
Other components
- Researcher
- WikiText-103
- CUDA GPU
Connections 38
- Researcher → Baseline training: starts baseline run
- Researcher → Adaptive training: starts adaptive run
- Researcher → Hard-skip evaluation: requests evaluation
- Researcher → Latency benchmark: requests benchmark
- Model configs → Baseline training: configures
- Model configs → Adaptive training: configures
- WikiText-103 → Dataset and batches: provides corpus
- Baseline training → Dataset and batches: prepares and batches
- Adaptive training → Dataset and batches: prepares and batches
- Baseline training → Baseline transformer: trains
- Adaptive training → Adaptive transformer: trains
- Adaptive training → Training losses: computes objective
- Adaptive transformer → Router designs: selects token gates
- Baseline training → Run checkpoints: saves run artifacts
- Adaptive training → Run checkpoints: saves run artifacts
- Run checkpoints → Hard-skip evaluation: supplies trained weights
- Hard-skip evaluation → Adaptive transformer: builds dense reference
- Hard-skip evaluation → Hard-skip inference: evaluates skipped model
- Hard-skip evaluation → Dataset and batches: samples validation batches
- Adaptive transformer → Hard-skip inference: provides reused blocks
- Run checkpoints → Latency benchmark: supplies trained weights
- Latency benchmark → Baseline transformer: times dense control
- Latency benchmark → Hard-skip inference: times skipped inference
- Latency benchmark → CUDA GPU: runs timed workload
- Hard-skip evaluation → Evaluation results: writes evaluation metrics
- Latency benchmark → Evaluation results: writes latency metrics
- Run checkpoints → Routing visualization: provides trained weights
- Routing visualization → Adaptive transformer: runs gate inference
- Routing visualization → Evaluation results: renders routing view
- Run checkpoints → Adaptive quality curve: reads run metrics
- Run checkpoints → Top-k quality curve: reads run metrics
- Top-k quality curve → Quality-speed chart: provides capacity losses
- Evaluation results → Quality-speed chart: provides benchmark latencies
- Quality-speed chart → Evaluation results: writes comparison chart
- Run checkpoints → Capacity diagnostics: provides trained weights
- Capacity diagnostics → Hard-skip inference: tests alternate capacities
- Capacity diagnostics → Dataset and batches: samples validation batches
- Capacity diagnostics → Evaluation results: writes diagnostic metrics