How internlm/sim-cot works
SIM-CoT is a research framework for training and evaluating language models that reason through implicit latent tokens, with step-level supervision intended to stabilize and make those latents interpretable. The README documents two execution paths: Coconut and CODI. In each, a runner prepares data and trains or evaluates a model; during training, an auxiliary decoder supervises latent reasoning steps, while inference omits that decoder. CODI also includes a latent-probing path for decoding and inspecting latent tokens. The graph reflects the sampled source and documented workflows; unsampled evaluation wiring is left general.
Main entry points: run.py Coconut runner · train.py CODI trainer · coconut.py Coconut model · probe_latent_token.py Latent probe
Components 14
Coconut path
- Coconut runner
Coconut/run.py - Latent datasets
Coconut/dataset.py - Coconut model
Coconut/coconut.py - Step decoder
Coconut/coconut.py
CODI path
- CODI trainer
CODI/train.py - Supervised examples
CODI/train.py - CODI model
CODI/src/model.py - Latent probe
CODI/probe_latent_token.py
Model and data
- Pretrained LM
- Reasoning data
Results
- Model checkpoints
- Decoded latent steps
- Evaluation results
Other components
- Researcher
Connections 17
- Researcher → Coconut runner: configures run
- Coconut runner → Latent datasets: builds datasets
- Coconut runner → Coconut model: trains or evaluates
- Latent datasets → Reasoning data: loads examples
- Coconut model → Pretrained LM: wraps language model
- Coconut model → Step decoder: supervises steps
- Coconut runner → Model checkpoints: saves model
- Coconut runner → Evaluation results: evaluates accuracy
- Researcher → CODI trainer: starts training
- CODI trainer → Supervised examples: formats examples
- CODI trainer → CODI model: trains model
- Supervised examples → Reasoning data: uses examples
- CODI model → Pretrained LM: loads language model
- CODI trainer → Model checkpoints: saves model
- Researcher → Latent probe: requests inspection
- Latent probe → CODI model: runs generation
- Latent probe → Decoded latent steps: decodes latents