# facebookresearch/online-dt architecture

> Architecture diagram of the GitHub repository facebookresearch/online-dt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/online-dt
- Repository: https://github.com/facebookresearch/online-dt

## Overview

##### Project type and purpose
- This is a research‐oriented machine‐learning codebase implementing the “Online Decision Transformer” in PyTorch.
- It orchestrates data downloading, preprocessing, model definition, training, evaluation, and logging for reinforcement‐learning experiments on D4RL benchmarks.

##### File structure
Top‐level files:
- main.py  → entry point: sets hyperparameters, instantiates components, kicks off training
- data.py  → dataset loading & preprocessing helpers
- data/download_… → scripts for grabbing D4RL antmaze & gym datasets
- decision_transformer/… → model definitions (generic “Model”, “DecisionTransformer”, GPT-style trajectory predictor)
- trainer.py  → training loop: sampling from buffer, forward/backward, optimization (Lamb)
- replay_buffer.py → experience storage & sampling interface
- evaluation.py → periodic validation/evaluation metrics
- lamb.py  → custom optimizer implementation
- logger.py  → tensorboard/file logging utilities
- utils.py  → miscellaneous helpers (e.g., seeding, config parsing)
- conda_env.yml → dependency spec

##### README for additional insights
- Confirms PyTorch, tensorboard, D4RL dependencies
- Shows usage: `python main.py` spawns an “exp” folder with logs and checkpoints
- Points to external D4RL data and required environment tweaks

a. Main components
- Data Layer
– D4RL Data Download Scripts
– Dataset Preprocessor (data.py)
– Replay Buffer
- Model Layer
– Base Model Interface (model.py)
– Decision Transformer (decision_transformer.py)
– GPT-style Trajectory Model (trajectory_gpt2.py)
- Training Layer
– Trainer (trainer.py)
– Optimizer (lamb.py)
- Evaluation Layer
– Evaluation Routine (evaluation.py)
- Logging Layer
– Logger (logger.py)
- Orchestration
– main.py drives configuration, instantiates all components, manages experiment loop

b. Relationships & interactions
1. main.py loads config → initializes Data Layer → populates Replay Buffer
2. Trainer requests batches from Replay Buffer, feeds to Model Layer
3. Model forward pass → loss → lamb optimizer updates parameters
4. Trainer triggers Evaluation periodically → evaluation.py consumes current model & buffer
5. Logger records training/eval metrics to disk/tensorboard

c. Architectural patterns & principles
- Modular separation: Data, Model, Training, Eval, Logging
- Inversion of control via main.py: central orchestrator
- Dependency injection: pass buffer & model references into Trainer
- Abstraction of optimizer and model interfaces for extensibility

d. Key technologies & frameworks
- PyTorch for neural network implementation
- D4RL (Gym) datasets for RL benchmarks
- Tensorboard for monitoring
- Custom Lamb optimizer
- Conda environment management

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Orchestration"
        Main["main.py"]:::orchestration
    end

    subgraph "Data Layer"
        Download["D4RL Download Scripts"]:::data
        Preprocess["Dataset Preprocessor"]:::data
        Buffer["Replay Buffer"]:::data
    end

    subgraph "Model Layer"
        ModelIntf["Base Model Interface"]:::model
        DT["Decision Transformer"]:::model
        GPT["GPT-style Trajectory Model"]:::model
    end

    subgraph "Training & Evaluation"
        TrainerNode["Trainer"]:::train
        Optim["Optimizer (Lamb)"]:::train
        Eval["Evaluation Routine"]:::train
    end

    subgraph "Logging Layer"
        LoggerNode["Logger"]:::logging
    end

    Main -->|"load config\ninstantiate Data Layer"| Download
    Download -->|"raw data"| Preprocess
    Preprocess -->|"processed data"| Buffer
    Main -->|"inject Buffer & Model"| TrainerNode
    TrainerNode -->|"requests batch"| Buffer
    Buffer -->|"batch of trajectories"| TrainerNode
    TrainerNode -->|"forward pass"| ModelIntf
    ModelIntf -->|"uses"| DT
    ModelIntf -->|"uses"| GPT
    DT -->|"predictions"| TrainerNode
    GPT -->|"predictions"| TrainerNode
    TrainerNode -->|"compute loss"| Optim
    Optim -->|"update params"| ModelIntf
    TrainerNode -->|"periodic eval"| Eval
    Eval -->|"evaluation metrics"| LoggerNode
    TrainerNode -->|"training metrics"| LoggerNode
    Main -->|"orchestrates loop"| TrainerNode

    click Main "https://github.com/facebookresearch/online-dt/blob/main/main.py"
    click Download "https://github.com/facebookresearch/online-dt/blob/main/data/download_d4rl_antmaze_datasets.py"
    click Preprocess "https://github.com/facebookresearch/online-dt/blob/main/data.py"
    click Buffer "https://github.com/facebookresearch/online-dt/blob/main/replay_buffer.py"
    click ModelIntf "https://github.com/facebookresearch/online-dt/blob/main/decision_transformer/models/model.py"
    click DT "https://github.com/facebookresearch/online-dt/blob/main/decision_transformer/models/decision_transformer.py"
    click GPT "https://github.com/facebookresearch/online-dt/blob/main/decision_transformer/models/trajectory_gpt2.py"
    click TrainerNode "https://github.com/facebookresearch/online-dt/blob/main/trainer.py"
    click Optim "https://github.com/facebookresearch/online-dt/blob/main/lamb.py"
    click Eval "https://github.com/facebookresearch/online-dt/blob/main/evaluation.py"
    click LoggerNode "https://github.com/facebookresearch/online-dt/blob/main/logger.py"

    classDef data fill:#D6EAF8,stroke:#3498DB;
    classDef model fill:#D5F5E3,stroke:#27AE60;
    classDef train fill:#FCF3CF,stroke:#F1C40F;
    classDef logging fill:#E8DAEF,stroke:#9B59B6;
    classDef orchestration fill:#FADBD8,stroke:#E74C3C;
```
