# kzl/decision-transformer architecture

> Architecture diagram of the GitHub repository kzl/decision-transformer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kzl/decision-transformer
- Repository: https://github.com/kzl/decision-transformer

## Overview

##### Project Type and Purpose
- This is a research codebase implementing “Decision Transformer: Reinforcement Learning via Sequence Modeling.”
- It contains two parallel experiment pipelines:
- Atari pipeline under `atari/` for game‐play experiments using the Arcade Learning Environment.
- Gym pipeline under `gym/` for continuous‐control tasks via OpenAI Gym and D4RL datasets.
- Each pipeline provides data preparation, model definitions, training scripts, and evaluation components.

##### File Structure
Top‐level:
- `atari/`
– `create_dataset.py`, `fixed_replay_buffer.py` → data collection & replay buffers
– `mingpt/` → GPT‐style model implementation (`model_atari.py`, `trainer_atari.py`, `utils.py`)
– `run.sh`, `run_dt_atari.py` → entry‐point scripts for launching experiments
– `conda_env.yml` → environment & dependencies
- `gym/`
– `data/` → scripts to download or preprocess D4RL datasets
– `decision_transformer/`
· `envs/` → custom Gym env wrappers/assets
· `models/` → core DT model, BC model, GPT‐based trajectory model
· `training/` → trainer classes for behavior cloning and sequence modeling
· `evaluation/` → evaluation scripts for running trained policies
– `experiment.py` → high‐level experiment orchestration
– `conda_env.yml` → environment & dependencies

##### README for Insights
- Confirms two directories correspond to two experiment tracks.
- Highlights the core architectural idea: framing RL as sequence modeling with a GPT backbone.

##### Main Components & Their Interactions

A. Data & Environment Layer
- For Atari: raw game frames → `fixed_replay_buffer.py` → dataset shards via `create_dataset.py`.
- For Gym: D4RL dataset downloader → data folder → preprocessed dataset objects.
- External: Arcade Learning Environment (ALE) or OpenAI Gym environments.

B. Model Layer
- GPT‐style transformer implementation (`mingpt/` or `decision_transformer/models/trajectory_gpt2.py`).
- Utility modules for tokenization, position embeddings, attention masks.

C. Training & Orchestration Layer
- Trainer classes (`trainer_atari.py`, `act_trainer.py`, `seq_trainer.py`)
- High‐level scripts: `run_dt_atari.py`, `experiment.py`
- They read datasets, instantiate the model, define optimization loop, log metrics.

D. Evaluation Layer
- Atari: evaluation via rollout scripts inside `trainer_atari.py` or custom evaluation utils.
- Gym: `evaluate_episodes.py` to load checkpoints, run episodes in envs, compute returns.

E. Environment & Dependency Configuration
- `conda_env.yml` in each directory
- Shell scripts (`run.sh`) for setting up PYTHONPATH and launching experiments

##### Architectural Patterns & Design Principles
- Modular separation of concerns: data pipeline, model definition, training logic, evaluation.
- Reusable transformer core (`mingpt` / `trajectory_gpt2`) across both pipelines.
- Clear experiment abstraction: user‐facing scripts orchestrate the common pipeline.
- Configuration‐driven (via YAML) for reproducibility.

##### Final Checklist
- Callouts for configuration files and environment setup.
- A title block referencing “Decision Transformer Architecture – Atari & Gym Pipelines.”

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data & Environment Layer"
        subgraph "Atari Pipeline"
            ALE["Arcade Learning Environment"]:::external
            FRB["Replay Buffer\n(fixed_replay_buffer.py)"]:::data
            CDS["Dataset Creation\n(create_dataset.py)"]:::data
            A_CF["Conda Env\n(conda_env.yml)"]:::config
            ALE -->|"store frames"| FRB
            FRB -->|"create shards"| CDS
        end
        subgraph "Gym Pipeline"
            GymEnv["OpenAI Gym & D4RL"]:::external
            DLD["Download D4RL Data\n(download_d4rl_datasets.py)"]:::data
            GPDS["Preprocessed Datasets"]:::data
            G_CF["Conda Env\n(conda_env.yml)"]:::config
            GymEnv -->|"access env"&nbsp;| DLD
            DLD -->|"download/preprocess"| GPDS
        end
    end

    subgraph "Model Layer"
        TCore["Transformer Core\n(mingpt/model_atari.py\ntrajectory_gpt2.py)"]:::model
        Utils["Model Utils\n(mingpt/utils.py)"]:::model
    end

    subgraph "Training & Orchestration Layer"
        subgraph "Atari Training"
            TrainerA["Atari Trainer\n(trainer_atari.py)"]:::train
            RunA["Run Script\n(run_dt_atari.py)"]:::train
            Shell["Runner Shell\n(run.sh)"]:::train
            CDS -->|"input data"| TrainerA
            TCore -->|"initialize"| TrainerA
            TrainerA -->|"launch"| RunA
            RunA -->|"via"| Shell
        end
        subgraph "Gym Training"
            TrainerAct["Act Trainer\n(act_trainer.py)"]:::train
            TrainerSeq["Seq Trainer\n(seq_trainer.py)"]:::train
            TrainerBase["Base Trainer\n(trainer.py)"]:::train
            Exp["Experiment\n(experiment.py)"]:::train
            GPDS -->|"input data"| TrainerAct
            GPDS -->|"input data"| TrainerSeq
            TCore -->|"initialize"| TrainerAct
            TCore -->|"initialize"| TrainerSeq
            TrainerAct -->|"extends"| TrainerBase
            TrainerSeq -->|"extends"| TrainerBase
            TrainerBase -->|"launch"| Exp
        end
    end

    subgraph "Evaluation Layer"
        EvalA["Atari Evaluation\n(in trainer_atari)"]:::eval
        EvalG["Gym Evaluation\n(evaluate_episodes.py)"]:::eval
        TrainerA -->|"checkpoint"| EvalA
        Exp -->|"checkpoint"| EvalG
    end

    %% Shared Data to Model
    CDS -->|"feeds"| TCore
    GPDS -->|"feeds"| TCore

    %% Click Events
    click FRB "https://github.com/kzl/decision-transformer/blob/master/atari/fixed_replay_buffer.py"
    click CDS "https://github.com/kzl/decision-transformer/blob/master/atari/create_dataset.py"
    click A_CF "https://github.com/kzl/decision-transformer/blob/master/atari/conda_env.yml"
    click DLD "https://github.com/kzl/decision-transformer/blob/master/gym/data/download_d4rl_datasets.py"
    click G_CF "https://github.com/kzl/decision-transformer/blob/master/gym/conda_env.yml"
    click TCore "https://github.com/kzl/decision-transformer/blob/master/atari/mingpt/model_atari.py"
    click Utils "https://github.com/kzl/decision-transformer/blob/master/atari/mingpt/utils.py"
    click TrainerA "https://github.com/kzl/decision-transformer/blob/master/atari/mingpt/trainer_atari.py"
    click RunA "https://github.com/kzl/decision-transformer/blob/master/atari/run_dt_atari.py"
    click Shell "https://github.com/kzl/decision-transformer/blob/master/atari/run.sh"
    click TrainerAct "https://github.com/kzl/decision-transformer/blob/master/gym/decision_transformer/training/act_trainer.py"
    click TrainerSeq "https://github.com/kzl/decision-transformer/blob/master/gym/decision_transformer/training/seq_trainer.py"
    click TrainerBase "https://github.com/kzl/decision-transformer/blob/master/gym/decision_transformer/training/trainer.py"
    click Exp "https://github.com/kzl/decision-transformer/blob/master/gym/experiment.py"
    click EvalG "https://github.com/kzl/decision-transformer/blob/master/gym/decision_transformer/evaluation/evaluate_episodes.py"

    classDef data fill:#9fd1ff,stroke:#000,color:#000
    classDef model fill:#9fff9f,stroke:#000,color:#000
    classDef train fill:#ffcf9f,stroke:#000,color:#000
    classDef eval fill:#df9fff,stroke:#000,color:#000
    classDef config fill:#e0e0e0,stroke:#000,color:#000
    classDef external fill:#cccccc,stroke:#000,stroke-dasharray: 5 5,color:#000
```
