How kzl/decision-transformer works
Project Type and Purpose
• This is a research codebase implementing “Decision Transformer: Reinforcement Learning via Sequence Modeling.”
• It contains two parallel experiment pipelines:
• Atari pipeline under atari/ for game‐play experiments using the Arcade Learning Environment.
• Gym pipeline under gym/ for continuous‐control tasks via OpenAI Gym and D4RL datasets.
• Each pipeline provides data preparation, model definitions, training scripts, and evaluation components.
File Structure
Top‐level:
• atari/
– create_dataset.py, fixed_replay_buffer.py → data collection & replay buffers
– mingpt/ → GPT‐style model implementation (model_atari.py, trainer_atari.py, utils.py)
– run.sh, run_dt_atari.py → entry‐point scripts for launching experiments
– conda_env.yml → environment & dependencies
• gym/
– data/ → scripts to download or preprocess D4RL datasets
– decision_transformer/
· envs/ → custom Gym env wrappers/assets
· models/ → core DT model, BC model, GPT‐based trajectory model
· training/ → trainer classes for behavior cloning and sequence modeling
· evaluation/ → evaluation scripts for running trained policies
– experiment.py → high‐level experiment orchestration
– conda_env.yml → environment & dependencies
README for Insights
• Confirms two directories correspond to two experiment tracks.
• Highlights the core architectural idea: framing RL as sequence modeling with a GPT backbone.
Main Components & Their Interactions
A. Data & Environment Layer
• For Atari: raw game frames → fixed_replay_buffer.py → dataset shards via create_dataset.py.
• For Gym: D4RL dataset downloader → data folder → preprocessed dataset objects.
• External: Arcade Learning Environment (ALE) or OpenAI Gym environments.
B. Model Layer
• GPT‐style transformer implementation (mingpt/ or decision_transformer/models/trajectory_gpt2.py).
• Utility modules for tokenization, position embeddings, attention masks.
C. Training & Orchestration Layer
• Trainer classes (trainer_atari.py, act_trainer.py, seq_trainer.py)
• High‐level scripts: run_dt_atari.py, experiment.py
• They read datasets, instantiate the model, define optimization loop, log metrics.
D. Evaluation Layer
• Atari: evaluation via rollout scripts inside trainer_atari.py or custom evaluation utils.
• Gym: evaluate_episodes.py to load checkpoints, run episodes in envs, compute returns.
E. Environment & Dependency Configuration
• conda_env.yml in each directory
• Shell scripts (run.sh) for setting up PYTHONPATH and launching experiments
Architectural Patterns & Design Principles
• Modular separation of concerns: data pipeline, model definition, training logic, evaluation.
• Reusable transformer core (mingpt / trajectory_gpt2) across both pipelines.
• Clear experiment abstraction: user‐facing scripts orchestrate the common pipeline.
• Configuration‐driven (via YAML) for reproducibility.
Final Checklist
• Callouts for configuration files and environment setup.
• A title block referencing “Decision Transformer Architecture – Atari & Gym Pipelines.”