How radixark/miles works
1. Top-level shape
- Miles is a Python-first RL/post-training framework with two main runtime planes: training and rollout/inference.
- The core packages are miles/ and miles_plugins/; the rest of the repo is orchestration (scripts/, examples/), packaging (docker/, tools/), and validation (tests/, CI).
2. Primary entrypoints
- train.py and train_async.py are the unified launch points for RL/SFT-style runs.
- They feed into shared argument parsing and utility layers under miles/utils/, then dispatch to the selected backend and rollout mode based on model, precision, and distributed topology.
3. Training backend boundary
- miles/backends/training_utils/ contains the training-side logic: data shaping, loss functions, advantage/reward handling, replay data, metrics, and distributed helpers.
- miles/backends/megatron_utils/ and miles/backends/experimental/fsdp_utils/ are the two main training implementations, with Megatron being the more feature-rich path for large-scale and low-precision runs.
4. Rollout / inference boundary
- miles/rollout/ owns the generation side: single-turn, multi-turn, tool-call, SFT-style rollout, sleep/idle rollout, and inference-rollout compatibility layers.
- This package is where prompt construction, sampling, logprob handling, filtering, and trajectory/session management happen before data is handed back to training.
5. SGLang integration
- miles/backends/sglang_utils/ and miles/rollout/sglang_rollout.py bridge the framework to SGLang for high-throughput serving and generation.
- Rollout data flows through SGLang-specific config/engine layers, then returns tokens, logprobs, and routing metadata needed by the training side.
6. Distributed orchestration
- miles/ray/ is the distributed control plane for rollout and training workers.
- It manages actor groups, placement, rollout servers, router management, address allocation, and server cells/groups, making Ray the coordination layer for multi-node execution.
7. Routing, replay, and mismatch handling
- miles/router/ and parts of miles/rollout/ support routing-aware execution, including the “R3” style replay path described in the docs.
- This is a key architectural boundary: inference records routing decisions, and training can replay or align them to reduce train-inference mismatch, especially for MoE models.
8. Model- and kernel-specific extensions
- miles_plugins/ contains model backends and custom kernels: DeepSeek, GLM, Qwen, Qwen-VL, MoE/indexer kernels, sparse MLA ops, and bridge logic.
- miles/backends/megatron_utils/megatron_to_hf/ and related update-weight code handle checkpoint conversion and parameter mapping between Megatron-style distributed weights and Hugging Face-style model layouts.
9. True on-policy and task-specific schemas
- miles/true_on_policy/ defines contracts, config, schemas, and model profiles for the “true on-policy” path.
- This package is a narrower abstraction layer used when the training loop needs stricter rollout/training consistency and explicit data contracts.
10. Utilities that support the whole stack
- miles/utils/ is a broad shared layer for async/process control, distributed helpers, chat templates/tokenization, logging, metrics, health monitoring, replay, and debugging.
- Several submodules are architecture-critical: chat-template utilities affect prompt/token parity, distributed utilities affect multi-node execution, and tracking utilities connect runs to TensorBoard, W&B, MLflow, or Prometheus.
11. Examples and recipe layer
- examples/ is not core runtime code; it is a catalog of runnable recipes for RL, SFT, LoRA, VLM, multi-agent, speculative decoding, search/tool use, and benchmark-specific setups.
- These examples show the intended composition pattern: data prep → rollout → reward/filtering → training → checkpoint/output conversion.
12. Model deployment and conversion tools
- tools/ and many scripts under scripts/ provide checkpoint conversion, precision conversion, and run-specific launch wrappers.
- These tools are important because the framework spans multiple checkpoint formats and precisions, so the architecture includes explicit conversion boundaries between HF, Megatron, distributed checkpoints, INT4/FP8/MXFP8/NVFP4 variants, and runtime-friendly artifacts.
13. Packaging, containerization, and CI
- docker/ contains the build images and backend patches for GPU/vendor-specific support, including NVIDIA, ROCm, and NPU-related patch sets.
- .github/workflows/, tests/, and tests/ci/ enforce that the distributed orchestration, kernels, model bridges, and rollout/training parity remain stable across supported environments.
14. End-to-end data flow to diagram
- A typical run starts in train.py or a recipe script, loads model/runtime config, launches rollout workers and training workers, generates trajectories through SGLang/Ray, applies reward/filter/loss logic, updates the model via Megatron/FSDP, then writes checkpoints and optionally converts them for inference or downstream serving.
- The most important diagram boundaries are: user scripts, rollout/inference plane, distributed orchestration plane, training plane, and model/plugin/kernel extensions.
Main entry points: rollout Rollout · training_utils Training · ray Ray · train.py train.py
Components 18
Entrypoints
Unified launch scripts and recipe wrappers
- train.py · Primary synchronous training entrypoint
train.py - train async · Asynchronous training entrypoint
train_async.py - Scripts · Model- and topology-specific launch wrappers
scripts
Core runtime
Shared framework packages for rollout, training, routing, and orchestration
- Utils · Argument parsing, async/process control, logging, metrics, tokenization, and shared helpers
miles/utils - Rollout · Prompting, sampling, sessions, filters, and rollout modes
miles/rollout - Training · Losses, data shaping, replay, metrics, and distributed training helpers
miles/backends/training_utils - Megatron · Feature-rich distributed training path and checkpoint/weight handling
miles/backends/megatron_utils - FSDP · Experimental FSDP training path
miles/backends/experimental/fsdp_utils - SGLang · Configuration and engine bridge for high-throughput generation
miles/backends/sglang_utils - Ray · Actors, placement, rollout servers, router management, and worker orchestration
miles/ray - Router · Routing-aware execution and replay alignment boundary
miles/router - On-policy · Contracts, configs, and schemas for strict rollout-training consistency
miles/true_on_policy
Extensions
Model-specific bridges, kernels, and plugins
- Plugins · Model bridges, custom kernels, and backend-specific extensions
miles_plugins
Ops & delivery
Packaging, containers, examples, tools, and validation
- Tools · Checkpoint and precision conversion utilities
tools - Examples · Runnable training, rollout, and evaluation recipes
examples - Docker · Container builds and vendor-specific patch sets
docker - Tests · Unit, integration, and end-to-end validation suites
tests - Docs · Architecture, user guide, examples, and model/platform docs
docs
Connections 26
- train.py → Utils: parse/configure · Loads shared args, runtime config, and utilities
- train async → Utils: parse/configure · Uses shared async and argument utilities
- train.py → Rollout: dispatch · Selects rollout mode and generation path
- train async → Rollout: dispatch · Runs async rollout/inference flow
- train.py → Training: train loop · Feeds trajectories into losses and optimization helpers
- train async → Training: train loop · Uses the same training-side data and loss logic
- Training → Megatron: backend · Megatron is the main large-scale implementation
- Training → FSDP: backend · Experimental FSDP implementation path
- Rollout → SGLang: serve/generate · Uses SGLang for high-throughput token generation
- Rollout → Ray: coordinate · Distributed rollout workers and servers are managed via Ray
- Ray → Rollout: data plane · Routes rollout requests through servers and sessions
- Ray → Training: workers · Coordinates training-side distributed actors and placement
- Rollout → Router: route-aware · Supports replay and mismatch-aware routing flows
- Router → Ray: orchestrate · Routing decisions are carried through distributed worker control
- On-policy → Training: contracts · Provides stricter data and model contracts for aligned training
- Plugins → Megatron: model kernels · Extends training with model-specific bridges and custom ops
- Plugins → SGLang: model bridges · Adapts model backends for serving/generation
- Tools → Megatron: convert · Supports checkpoint and parameter mapping workflows
- Examples → train.py: invoke · Recipe scripts launch the main training entrypoint
- Examples → Rollout: compose · Examples demonstrate rollout and evaluation patterns
- Docker → Plugins: patches · Container builds carry vendor and kernel-specific patches
- Tests → Ray: validate · Covers distributed orchestration and rollout server behavior
- Tests → Rollout: validate · Covers generation, sessions, filters, and inference rollout
- Tests → Training: validate · Covers loss, replay, and training-side numerics
- Docs → Examples: describe · Documents the intended recipes and workflows
- Docs → Router: describe · Documents routing and mismatch mitigation concepts