# ra9hur/decision-transformers-for-trading architecture

> Architecture diagram of the GitHub repository ra9hur/decision-transformers-for-trading, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ra9hur/decision-transformers-for-trading
- Repository: https://github.com/ra9hur/decision-transformers-for-trading

## Overview

##### Project Type and Purpose
- This is an offline‐Reinforcement‐Learning (RL) pipeline built in Jupyter notebooks, which:
– Collects financial data via FinRL (YahooDownloader)
– Generates PPO trajectories in a Gym-anytrading environment
– Prepares sequence datasets for a Decision Transformer (DT) model
– Trains the DT using HuggingFace-style Trainer
– Evaluates and visualizes performance against PPO, MVO, and DJI benchmarks

##### Main Architectural Components
a. Data Ingestion & Storage
– External Service: Yahoo Finance API (via FinRL downloader)
– Raw data files: CSV/JSON in `/data/dataset`
– Preprocessed pickles: `df_actions_*.pkl`, `df_account_value_*.pkl`, etc.
b. Trajectory Generation
– Gym-anytrading Environment
– PPO Agent (pretrained or loaded from `trained_models/agent_ppo.zip`)
– Outputs: sequences of (state, action, reward, done) stored as pickles
c. Dataset Preparation
– Notebooks “2_prepare_inputs_to_DT.ipynb”
– Collator code: batches up context windows, attention masks, returns, timesteps
– Input for Trainer: dictionary with `loss`, action targets, masks
d. Model Training
– TrainableDT class inherits Decision Transformer
– HuggingFace Trainer instantiates model with `training_args.bin` and `config.json`
– Checkpoints/artifacts written to `/trained_models` (e.g. `model.safetensors`)
e. Evaluation
– Notebook “4_evaluate_DT.ipynb”
– Loads trained model, seeds environment, runs inference on held-out data
– Computes metrics: cumulative returns, Sharpe ratio, etc.
f. Visualization
– Notebook “5_visualize_performanmce_dt.ipynb”
– Generates comparison plots (DT vs PPO vs DJI vs MVO)
– Exports figures to `/images`

##### Relationships & Data Flows
- Data flows left to right:
[Yahoo API] → [Raw Data Store] → [Env + PPO Agent] → [Trajectory Store] →
[Dataset Prep] → [Decision Transformer Trainer] → [Model Artifacts] →
[Evaluation] → [Visualization Outputs]

##### Important Patterns & Principles
- Modular pipelines in Jupyter notebooks – each cell is a micro-step
- Separation of concerns: data layer, compute layer (training), evaluation layer
- Offline RL pattern: fixed dataset feeds generative sequence model
- Reuse of pretrained PPO as data generator, then generative modeling with DT

##### Technologies & Frameworks
- FinRL (YahooDownloader, preprocessing tools)
- Gym-anytrading for environment simulation
- PyTorch & HuggingFace Transformers for Decision Transformer
- HuggingFace Trainer API for abstracted training loop
- Jupyter notebooks for orchestration & visualization
- pickle/CSV/JSON for data interchange

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion & Storage
    subgraph "Data Ingestion & Storage"
        Y("Yahoo Finance API"):::external
        NB1["1_collect_dataset.ipynb"]:::notebook
        Raw["Raw Data Store"]:::data-store
        subgraph "Dataset Files"
            ds1["dataset_info.json"]:::data-store
            ds2["state.json"]:::data-store
        end
    end

    %% Preprocessed Pickles
    subgraph "Preprocessed Pickles"
        p1["df_account_value_dt.pkl"]:::data-store
        p2["df_account_value_ppo.pkl"]:::data-store
        p3["df_actions_dt.pkl"]:::data-store
        p4["df_actions_ppo.pkl"]:::data-store
        p5["trade.csv"]:::data-store
        p6["train.csv"]:::data-store
    end

    %% Trajectory Generation
    subgraph "Trajectory Generation"
        Env["Gym-anytrading Environment"]:::compute
        PPOAgent["PPO Agent"]:::compute
        Traj["Trajectory Store"]:::data-store
    end

    %% Dataset Preparation
    DP["2_prepare_inputs_to_DT.ipynb"]:::notebook

    %% Model Training
    subgraph "Model Training"
        NB3["3_Train_DT.ipynb"]:::notebook
        Trainer["HuggingFace Trainer"]:::compute
        subgraph "Model Artifacts"
            cfg["config.json"]:::data-store
            args["training_args.bin"]:::data-store
            model["model.safetensors"]:::data-store
        end
    end

    %% Evaluation
    NB4["4_evaluate_DT.ipynb"]:::notebook
    EvalRes["Evaluation Results"]:::compute

    %% Visualization
    NB5["5_visualize_performanmce_dt.ipynb"]:::notebook
    Out["Performance Figures"]:::data-store

    %% Dependencies
    subgraph "External Dependencies"
        FinRL["FinRL"]:::external
        GymDep["Gym-anytrading"]:::external
        HF["HuggingFace Transformers"]:::external
    end

    %% Connections
    FinRL --> NB1
    Y -->|"download"| NB1
    NB1 -->|"save raw"| Raw
    Raw --> ds1
    Raw --> ds2
    NB1 -->|"preprocess"| p1
    NB1 --> p2
    NB1 --> p3
    NB1 --> p4
    NB1 --> p5
    NB1 --> p6

    p1 --> Env
    p2 --> Env
    p3 --> Env
    p4 --> Env
    PPOAgent -->|"generate"| Traj
    Env --> Traj

    Traj --> DP
    DP -->|"prepare dataset"| Trainer

    NB3 --> Trainer
    DP --> Trainer
    Trainer --> cfg
    Trainer --> args
    Trainer --> model

    model --> NB4
    NB4 --> EvalRes

    EvalRes --> NB5
    NB5 --> Out

    GymDep --> Env
    HF --> Trainer

    %% Click Events
    click NB1 "https://github.com/ra9hur/decision-transformers-for-trading/blob/main/1_collect_dataset.ipynb"
    click ds1 "https://github.com/ra9hur/decision-transformers-for-trading/blob/main/data/dataset/dataset_info.json"
    click ds2 "https://github.com/ra9hur/decision-transformers-for-trading/blob/main/data/dataset/state.json"
    click p1 "https://github.com/ra9hur/decision-transformers-for-trading/blob/main/data/df_account_value_dt.pkl"
    click p2 "https://github.com/ra9hur/decision-transformers-for-trading/blob/main/data/df_account_value_ppo.pkl"
    click p3 "https://github.com/ra9hur/decision-transformers-for-trading/blob/main/data/df_actions_dt.pkl"
    click p4 "https://github.com/ra9hur/decision-transformers-for-trading/blob/main/data/df_actions_ppo.pkl"
    click p5 "https://github.com/ra9hur/decision-transformers-for-trading/blob/main/data/trade.csv"
    click p6 "https://github.com/ra9hur/decision-transformers-for-trading/blob/main/data/train.csv"
    click PPOAgent "https://github.com/ra9hur/decision-transformers-for-trading/blob/main/trained_models/agent_ppo.zip"
    click DP "https://github.com/ra9hur/decision-transformers-for-trading/blob/main/2_prepare_inputs_to_DT.ipynb"
    click NB3 "https://github.com/ra9hur/decision-transformers-for-trading/blob/main/3_Train_DT.ipynb"
    click cfg "https://github.com/ra9hur/decision-transformers-for-trading/blob/main/trained_models/config.json"
    click args "https://github.com/ra9hur/decision-transformers-for-trading/blob/main/trained_models/training_args.bin"
    click model "https://github.com/ra9hur/decision-transformers-for-trading/blob/main/trained_models/model.safetensors"
    click NB4 "https://github.com/ra9hur/decision-transformers-for-trading/blob/main/4_evaluate_DT.ipynb"
    click NB5 "https://github.com/ra9hur/decision-transformers-for-trading/blob/main/5_visualize_performanmce_dt.ipynb"
    click Out "https://github.com/ra9hur/decision-transformers-for-trading/tree/main/images"

    %% Styles
    classDef data-store fill:#ccf,stroke:#333,stroke-width:1px
    classDef compute fill:#cfc,stroke:#333,stroke-width:1px
    classDef external fill:#eee,stroke:#333,stroke-width:1px
    classDef notebook fill:#ffc,stroke:#333,stroke-width:1px
```
