# adithya916/reinforcement-learning-market-making-agent architecture

> Architecture diagram of the GitHub repository adithya916/reinforcement-learning-market-making-agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/adithya916/reinforcement-learning-market-making-agent
- Repository: https://github.com/adithya916/reinforcement-learning-market-making-agent

## Overview

This is a compact PPO market-making research framework: a Gym environment simulates price movement and fills, agents choose quoting spreads, and the runner trains PPO before comparing it with an Avellaneda–Stoikov baseline. The graph separates environment dynamics, agent internals, and benchmark orchestration. The README documents the overall workflow; sampled source shows the concrete calls. Environment stages are conceptual responsibilities within one file, not separate runtime modules.

Main entry points: [`main_run.py`](https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/main_run.py) (Workflow Runner), [`ppo_agent.py`](https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/ppo_agent.py) (PPO Agent), [`mm_env.py`](https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/mm_env.py) (Market Environment)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_environment["Market Environment"]
  node_environment["Market Environment<br/>[mm_env.py]"]
  node_price["Price and Volatility<br/>[mm_env.py]"]
  node_execution["Order Execution<br/>[mm_env.py]"]
  node_reward["PnL and Reward<br/>[mm_env.py]"]
end

subgraph group_agents["Trading Agents"]
  node_ppo["PPO Agent<br/>[ppo_agent.py]"]
  node_actorcritic["Actor-Critic Policy<br/>[ppo_agent.py]"]
  node_buffer[("Transition Buffer<br/>[ppo_agent.py]")]
  node_baseline["AS Baseline<br/>[ppo_agent.py]"]
end

subgraph group_benchmark["Training and Benchmark"]
  node_runner["Workflow Runner<br/>[main_run.py]"]
  node_trainer["PPO Training<br/>[main_run.py]"]
  node_rollouts["Validation Rollouts<br/>[main_run.py]"]
  node_metrics["Performance Metrics<br/>[main_run.py]"]
  node_report["Comparison Report<br/>[main_run.py]"]
end

node_user(("User"))

node_user -->|"runs"| node_runner
node_runner -->|"starts training"| node_trainer
node_trainer -->|"steps"| node_environment
node_trainer -->|"selects actions"| node_ppo
node_trainer -->|"stores transitions"| node_ppo
node_trainer -->|"updates policy"| node_ppo
node_runner -->|"creates"| node_baseline
node_runner -->|"runs validation"| node_rollouts
node_rollouts -->|"steps"| node_environment
node_rollouts -->|"selects actions"| node_ppo
node_rollouts -->|"gets actions"| node_baseline
node_rollouts -->|"provides PnL and inventory"| node_metrics
node_runner -->|"calculates"| node_metrics
node_runner -->|"prints"| node_report
node_environment -->|"updates"| node_price
node_environment -->|"handles fills"| node_execution
node_environment -->|"computes reward"| node_reward
node_ppo -->|"samples and evaluates"| node_actorcritic
node_ppo -->|"stores transitions"| node_buffer
node_baseline -->|"maps state to spread"| node_baseline

click node_runner "https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/main_run.py"
click node_trainer "https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/main_run.py"
click node_rollouts "https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/main_run.py"
click node_metrics "https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/main_run.py"
click node_report "https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/main_run.py"
click node_environment "https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/mm_env.py"
click node_price "https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/mm_env.py"
click node_execution "https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/mm_env.py"
click node_reward "https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/mm_env.py"
click node_ppo "https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/ppo_agent.py"
click node_actorcritic "https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/ppo_agent.py"
click node_buffer "https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/ppo_agent.py"
click node_baseline "https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/ppo_agent.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_environment,node_price,node_execution,node_reward,node_user toneBlue
class node_ppo,node_actorcritic,node_buffer,node_baseline toneAmber
class node_runner,node_trainer,node_rollouts,node_metrics,node_report toneMint
```

## Components

### Market Environment

- [Market Environment](https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/mm_env.py): `mm_env.py`
- [Price and Volatility](https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/mm_env.py): `mm_env.py`
- [Order Execution](https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/mm_env.py): `mm_env.py`
- [PnL and Reward](https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/mm_env.py): `mm_env.py`

### Trading Agents

- [PPO Agent](https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/ppo_agent.py): `ppo_agent.py`
- [Actor-Critic Policy](https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/ppo_agent.py): `ppo_agent.py`
- [Transition Buffer](https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/ppo_agent.py): `ppo_agent.py`
- [AS Baseline](https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/ppo_agent.py): `ppo_agent.py`

### Training and Benchmark

- [Workflow Runner](https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/main_run.py): `main_run.py`
- [PPO Training](https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/main_run.py): `main_run.py`
- [Validation Rollouts](https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/main_run.py): `main_run.py`
- [Performance Metrics](https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/main_run.py): `main_run.py`
- [Comparison Report](https://github.com/adithya916/reinforcement-learning-market-making-agent/blob/main/main_run.py): `main_run.py`

### Other components

- **User**

## Connections

- User → Workflow Runner: runs (evidence: `README.md`)
- Workflow Runner → PPO Training: starts training (evidence: `main_run.py`)
- PPO Training → Market Environment: steps (evidence: `main_run.py`)
- PPO Training → PPO Agent: selects actions (evidence: `main_run.py`)
- PPO Training → PPO Agent: stores transitions (evidence: `main_run.py`)
- PPO Training → PPO Agent: updates policy (evidence: `main_run.py`)
- Workflow Runner → AS Baseline: creates (evidence: `main_run.py`)
- Workflow Runner → Validation Rollouts: runs validation (evidence: `main_run.py`)
- Validation Rollouts → Market Environment: steps (evidence: `main_run.py`)
- Validation Rollouts → PPO Agent: selects actions (evidence: `main_run.py`)
- Validation Rollouts → AS Baseline: gets actions (evidence: `main_run.py`)
- Validation Rollouts → Performance Metrics: provides PnL and inventory (evidence: `main_run.py`)
- Workflow Runner → Performance Metrics: calculates (evidence: `main_run.py`)
- Workflow Runner → Comparison Report: prints (evidence: `main_run.py`)
- Market Environment → Price and Volatility: updates (evidence: `mm_env.py`)
- Market Environment → Order Execution: handles fills (evidence: `mm_env.py`)
- Market Environment → PnL and Reward: computes reward (evidence: `mm_env.py`)
- PPO Agent → Actor-Critic Policy: samples and evaluates (evidence: `ppo_agent.py`)
- PPO Agent → Transition Buffer: stores transitions (evidence: `ppo_agent.py`)
- AS Baseline → AS Baseline: maps state to spread (evidence: `ppo_agent.py`)
