# thesaifhub/adaptive-compute-transformer architecture

> Architecture diagram of the GitHub repository thesaifhub/adaptive-compute-transformer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thesaifhub/adaptive-compute-transformer
- Repository: https://github.com/thesaifhub/adaptive-compute-transformer

## Overview

Adaptive-Compute Transformer is a research system for training token-routing decoder transformers and testing whether reduced compute preserves language-model quality and improves real inference latency. The main workflow prepares WikiText-103 data, trains baseline or adaptive models, then evaluates adaptive checkpoints with hard-skip inference and benchmarks their latency against a dense baseline. Analysis scripts compare quality and speed across router budgets and visualize token routing. The excerpts show key model, training, hard-skip, and benchmark behavior; links for unsampled data and analysis modules follow the README and repository tree.

Main entry points: [`experiments`](https://github.com/thesaifhub/adaptive-compute-transformer/tree/main/experiments) (Run checkpoints), [`src/train_adaptive.py`](https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/train_adaptive.py) (Adaptive training), [`src/eval_hard_skip.py`](https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/eval_hard_skip.py) (Hard-skip evaluation), [`src/benchmark_speed.py`](https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/benchmark_speed.py) (Latency benchmark)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data and configuration"]
  node_config["Model configs<br/>[config.py]"]
  node_data_pipeline["Dataset and batches<br/>[data.py]"]
end

subgraph group_model["Transformer and routing"]
  node_baseline["Baseline transformer<br/>[model.py]"]
  node_adaptive["Adaptive transformer<br/>[model.py]"]
  node_router["Router designs<br/>[router.py]"]
  node_losses["Training losses<br/>[losses.py]"]
end

subgraph group_experiments["Training and evaluation"]
  node_baseline_training["Baseline training<br/>[train.py]"]
  node_adaptive_training["Adaptive training<br/>[train_adaptive.py]"]
  node_checkpoints["Run checkpoints"]
  node_hard_skip["Hard-skip inference<br/>[hard_skip.py]"]
  node_hard_skip_eval["Hard-skip evaluation<br/>[eval_hard_skip.py]"]
end

subgraph group_analysis["Measurement and analysis"]
  node_benchmark["Latency benchmark<br/>[benchmark_speed.py]"]
  node_diagnostics["Capacity diagnostics"]
  node_routing_visualization["Routing visualization"]
  node_adaptive_curve["Adaptive quality curve<br/>[plot_curve.py]"]
  node_topk_curve["Top-k quality curve<br/>[plot_topk_curve.py]"]
  node_combined_chart["Quality-speed chart"]
  node_results["Evaluation results"]
end

node_researcher(("Researcher"))
node_wikitext[("WikiText-103")]
node_gpu(("CUDA GPU"))

node_researcher -->|"starts baseline run"| node_baseline_training
node_researcher -->|"starts adaptive run"| node_adaptive_training
node_researcher -->|"requests evaluation"| node_hard_skip_eval
node_researcher -->|"requests benchmark"| node_benchmark
node_config -->|"configures"| node_baseline_training
node_config -->|"configures"| node_adaptive_training
node_wikitext -->|"provides corpus"| node_data_pipeline
node_baseline_training -->|"prepares and batches"| node_data_pipeline
node_adaptive_training -->|"prepares and batches"| node_data_pipeline
node_baseline_training -->|"trains"| node_baseline
node_adaptive_training -->|"trains"| node_adaptive
node_adaptive_training -->|"computes objective"| node_losses
node_adaptive -->|"selects token gates"| node_router
node_baseline_training -->|"saves run artifacts"| node_checkpoints
node_adaptive_training -->|"saves run artifacts"| node_checkpoints
node_checkpoints -->|"supplies trained weights"| node_hard_skip_eval
node_hard_skip_eval -->|"builds dense reference"| node_adaptive
node_hard_skip_eval -->|"evaluates skipped model"| node_hard_skip
node_hard_skip_eval -->|"samples validation batches"| node_data_pipeline
node_adaptive -->|"provides reused blocks"| node_hard_skip
node_checkpoints -->|"supplies trained weights"| node_benchmark
node_benchmark -->|"times dense control"| node_baseline
node_benchmark -->|"times skipped inference"| node_hard_skip
node_benchmark -.->|"runs timed workload"| node_gpu
node_hard_skip_eval -->|"writes evaluation metrics"| node_results
node_benchmark -->|"writes latency metrics"| node_results
node_checkpoints -->|"provides trained weights"| node_routing_visualization
node_routing_visualization -->|"runs gate inference"| node_adaptive
node_routing_visualization -->|"renders routing view"| node_results
node_checkpoints -->|"reads run metrics"| node_adaptive_curve
node_checkpoints -->|"reads run metrics"| node_topk_curve
node_topk_curve -->|"provides capacity losses"| node_combined_chart
node_results -->|"provides benchmark latencies"| node_combined_chart
node_combined_chart -->|"writes comparison chart"| node_results
node_checkpoints -->|"provides trained weights"| node_diagnostics
node_diagnostics -->|"tests alternate capacities"| node_hard_skip
node_diagnostics -->|"samples validation batches"| node_data_pipeline
node_diagnostics -->|"writes diagnostic metrics"| node_results

click node_config "https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/config.py"
click node_data_pipeline "https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/data.py"
click node_baseline "https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/model.py"
click node_adaptive "https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/model.py"
click node_router "https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/router.py"
click node_losses "https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/losses.py"
click node_baseline_training "https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/train.py"
click node_adaptive_training "https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/train_adaptive.py"
click node_checkpoints "https://github.com/thesaifhub/adaptive-compute-transformer/tree/main/experiments"
click node_hard_skip "https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/hard_skip.py"
click node_hard_skip_eval "https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/eval_hard_skip.py"
click node_benchmark "https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/benchmark_speed.py"
click node_diagnostics "https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/diagnose_offcapacity.py"
click node_routing_visualization "https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/visualize_routing.py"
click node_adaptive_curve "https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/plot_curve.py"
click node_topk_curve "https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/plot_topk_curve.py"
click node_combined_chart "https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/plot_speed_vs_accuracy.py"
click node_results "https://github.com/thesaifhub/adaptive-compute-transformer/tree/main/results"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_config,node_data_pipeline toneBlue
class node_baseline,node_adaptive,node_router,node_losses,node_wikitext toneAmber
class node_baseline_training,node_adaptive_training,node_checkpoints,node_hard_skip,node_hard_skip_eval toneMint
class node_benchmark,node_diagnostics,node_routing_visualization,node_adaptive_curve,node_topk_curve,node_combined_chart,node_results toneRose
class node_researcher,node_gpu toneIndigo
```

## Components

### Data and configuration

- [Model configs](https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/config.py): `src/config.py`
- [Dataset and batches](https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/data.py): `src/data.py`

### Transformer and routing

- [Baseline transformer](https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/model.py): `src/model.py`
- [Adaptive transformer](https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/model.py): `src/model.py`
- [Router designs](https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/router.py): `src/router.py`
- [Training losses](https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/losses.py): `src/losses.py`

### Training and evaluation

- [Baseline training](https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/train.py): `src/train.py`
- [Adaptive training](https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/train_adaptive.py): `src/train_adaptive.py`
- [Run checkpoints](https://github.com/thesaifhub/adaptive-compute-transformer/tree/main/experiments): `experiments`
- [Hard-skip inference](https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/hard_skip.py): `src/hard_skip.py`
- [Hard-skip evaluation](https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/eval_hard_skip.py): `src/eval_hard_skip.py`

### Measurement and analysis

- [Latency benchmark](https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/benchmark_speed.py): `src/benchmark_speed.py`
- [Capacity diagnostics](https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/diagnose_offcapacity.py): `src/diagnose_offcapacity.py`
- [Routing visualization](https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/visualize_routing.py): `src/visualize_routing.py`
- [Adaptive quality curve](https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/plot_curve.py): `src/plot_curve.py`
- [Top-k quality curve](https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/plot_topk_curve.py): `src/plot_topk_curve.py`
- [Quality-speed chart](https://github.com/thesaifhub/adaptive-compute-transformer/blob/main/src/plot_speed_vs_accuracy.py): `src/plot_speed_vs_accuracy.py`
- [Evaluation results](https://github.com/thesaifhub/adaptive-compute-transformer/tree/main/results): `results`

### Other components

- **Researcher**
- **WikiText-103**
- **CUDA GPU**

## Connections

- Researcher → Baseline training: starts baseline run
- Researcher → Adaptive training: starts adaptive run
- Researcher → Hard-skip evaluation: requests evaluation
- Researcher → Latency benchmark: requests benchmark
- Model configs → Baseline training: configures
- Model configs → Adaptive training: configures
- WikiText-103 → Dataset and batches: provides corpus
- Baseline training → Dataset and batches: prepares and batches
- Adaptive training → Dataset and batches: prepares and batches
- Baseline training → Baseline transformer: trains
- Adaptive training → Adaptive transformer: trains
- Adaptive training → Training losses: computes objective
- Adaptive transformer → Router designs: selects token gates
- Baseline training → Run checkpoints: saves run artifacts
- Adaptive training → Run checkpoints: saves run artifacts
- Run checkpoints → Hard-skip evaluation: supplies trained weights
- Hard-skip evaluation → Adaptive transformer: builds dense reference
- Hard-skip evaluation → Hard-skip inference: evaluates skipped model
- Hard-skip evaluation → Dataset and batches: samples validation batches
- Adaptive transformer → Hard-skip inference: provides reused blocks
- Run checkpoints → Latency benchmark: supplies trained weights
- Latency benchmark → Baseline transformer: times dense control
- Latency benchmark → Hard-skip inference: times skipped inference
- Latency benchmark → CUDA GPU: runs timed workload
- Hard-skip evaluation → Evaluation results: writes evaluation metrics
- Latency benchmark → Evaluation results: writes latency metrics
- Run checkpoints → Routing visualization: provides trained weights
- Routing visualization → Adaptive transformer: runs gate inference
- Routing visualization → Evaluation results: renders routing view
- Run checkpoints → Adaptive quality curve: reads run metrics
- Run checkpoints → Top-k quality curve: reads run metrics
- Top-k quality curve → Quality-speed chart: provides capacity losses
- Evaluation results → Quality-speed chart: provides benchmark latencies
- Quality-speed chart → Evaluation results: writes comparison chart
- Run checkpoints → Capacity diagnostics: provides trained weights
- Capacity diagnostics → Hard-skip inference: tests alternate capacities
- Capacity diagnostics → Dataset and batches: samples validation batches
- Capacity diagnostics → Evaluation results: writes diagnostic metrics