How microsoft/smart works
SMART is an offline reinforcement-learning research system: it trains a multi-task control transformer on DeepMind Control replay data, then fine-tunes a policy for a selected downstream task. The graph follows the documented pretraining and downstream workflows and includes the replay/data, transformer, and environment layers. Some configuration-driven class selection is not visible in the excerpts, so those links are documented from the README rather than inferred from imports.
Main entry points: dmc_downstream.py Downstream CLI · dmc_pretrain.py Pretrain CLI · dmc_datamodule.py DMC data modules · multitask_ct_module.py Multi-task transformer
Components 18
Training workflows
- Pretrain CLI
src/dmc_pretrain.py - Downstream CLI
src/dmc_downstream.py - Lightning trainer
src/utils/pl_utils.py - Pretrain config
configs/pretrain.yaml - Downstream config
configs/downstream.yaml
Offline data
- Replay datasets
src/datamodules/dmc_datamodule.py - DMC data modules
src/datamodules/dmc_datamodule.py - Replay buffer
src/datamodules/dmc_replay_buffer.py - Test dataset
src/datamodules/dummy_datamodule.py
Control models
- Multi-task transformer
src/models/multitask_ct_module.py - Control transformer
src/models/ct_module.py - GPT backbone
src/models/components/mingpt.py - Model utilities
src/models/utils.py
Environment runtime
- DMC environment factory
src/envs/dmc2gym/dmc2gym/__init__.py - Gym wrapper
src/envs/dmc2gym/dmc2gym/wrappers.py - DeepMind Control
Other components
- Researcher
- Azure data storage
Connections 22
- Researcher → Pretrain CLI: runs pretraining
README.md - Researcher → Downstream CLI: runs fine-tuning
README.md - Pretrain config → Pretrain CLI: configures
README.md - Downstream config → Downstream CLI: configures
README.md - Pretrain CLI → Lightning trainer: creates trainer
src/dmc_pretrain.py - Downstream CLI → Lightning trainer: creates trainer
src/dmc_downstream.py - Pretrain CLI → Multi-task transformer: instantiates model
src/dmc_pretrain.py - Pretrain CLI → Test dataset: tests with
src/dmc_pretrain.py - Downstream CLI → DMC data modules: creates data module
src/dmc_downstream.py - Downstream CLI → Test dataset: tests with
src/dmc_downstream.py - Replay datasets → Azure data storage: downloaded from
README.md - DMC data modules → Replay buffer: uses buffer
src/datamodules/dmc_datamodule.py - DMC data modules → DMC environment factory: creates environments
src/datamodules/dmc_datamodule.py - DMC environment factory → Gym wrapper: registers wrapper
src/envs/dmc2gym/dmc2gym/__init__.py - Gym wrapper → DeepMind Control: loads tasks
src/envs/dmc2gym/dmc2gym/wrappers.py - Multi-task transformer → GPT backbone: uses backbone
src/models/multitask_ct_module.py - Control transformer → GPT backbone: uses backbone
src/models/ct_module.py - Multi-task transformer → Model utilities: gets action sizes
src/models/multitask_ct_module.py - Control transformer → Model utilities: gets action sizes
src/models/ct_module.py - DMC data modules → Model utilities: uses action helpers
src/datamodules/dmc_datamodule.py - Downstream CLI → Control transformer: selects configured model
- Multi-task transformer → DMC environment factory: evaluates policy