How ra9hur/decision-transformers-for-trading works
Project Type and Purpose
• This is an offline‐Reinforcement‐Learning (RL) pipeline built in Jupyter notebooks, which:
– Collects financial data via FinRL (YahooDownloader)
– Generates PPO trajectories in a Gym-anytrading environment
– Prepares sequence datasets for a Decision Transformer (DT) model
– Trains the DT using HuggingFace-style Trainer
– Evaluates and visualizes performance against PPO, MVO, and DJI benchmarks
Main Architectural Components
a. Data Ingestion & Storage
– External Service: Yahoo Finance API (via FinRL downloader)
– Raw data files: CSV/JSON in /data/dataset
– Preprocessed pickles: df_actions_*.pkl, df_account_value_*.pkl, etc.
b. Trajectory Generation
– Gym-anytrading Environment
– PPO Agent (pretrained or loaded from trained_models/agent_ppo.zip)
– Outputs: sequences of (state, action, reward, done) stored as pickles
c. Dataset Preparation
– Notebooks “2_prepare_inputs_to_DT.ipynb”
– Collator code: batches up context windows, attention masks, returns, timesteps
– Input for Trainer: dictionary with loss, action targets, masks
d. Model Training
– TrainableDT class inherits Decision Transformer
– HuggingFace Trainer instantiates model with training_args.bin and config.json
– Checkpoints/artifacts written to /trained_models (e.g. model.safetensors)
e. Evaluation
– Notebook “4_evaluate_DT.ipynb”
– Loads trained model, seeds environment, runs inference on held-out data
– Computes metrics: cumulative returns, Sharpe ratio, etc.
f. Visualization
– Notebook “5_visualize_performanmce_dt.ipynb”
– Generates comparison plots (DT vs PPO vs DJI vs MVO)
– Exports figures to /images
Relationships & Data Flows
• Data flows left to right:
[Yahoo API] → [Raw Data Store] → [Env + PPO Agent] → [Trajectory Store] →
[Dataset Prep] → [Decision Transformer Trainer] → [Model Artifacts] →
[Evaluation] → [Visualization Outputs]
Important Patterns & Principles
• Modular pipelines in Jupyter notebooks – each cell is a micro-step
• Separation of concerns: data layer, compute layer (training), evaluation layer
• Offline RL pattern: fixed dataset feeds generative sequence model
• Reuse of pretrained PPO as data generator, then generative modeling with DT
Technologies & Frameworks
• FinRL (YahooDownloader, preprocessing tools)
• Gym-anytrading for environment simulation
• PyTorch & HuggingFace Transformers for Decision Transformer
• HuggingFace Trainer API for abstracted training loop
• Jupyter notebooks for orchestration & visualization
• pickle/CSV/JSON for data interchange