# s-marton/sympol architecture

> Architecture diagram of the GitHub repository s-marton/sympol, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/s-marton/sympol
- Repository: https://github.com/s-marton/sympol

## Overview

##### Project Type and Purpose
- Type: A research-grade on-policy actor-critic reinforcement-learning framework, specialized in learning symbolic decision-tree policies (SYMPOL) with PPO.
- Purpose: Provide an end-to-end pipeline for training interpretable tree-based actors alongside neural critics, with optional hyperparameter optimization and logging.

##### File Structure
Top-level modules:
- args.py            – command-line interface and argument parsing
- configs.py         – hyperparameter definitions and Optuna search space
- utils.py           – shared helper functions (seeding, logging setup)
- sympol.py          – implementation of the symbolic decision-tree actor
- sdt.py             – discretized/soft decision-tree baselines
- mlp.py             – multi-layer perceptron critic
- ppo.py             – main training script for Gymnasium environments
- ppo_gymnax.py      – main training script for vectorized Gymnax environments
- run_ppo.sh         – shell helper for parallel hyperparameter studies
- commands_to_run.txt– exact experiment commands for reproducibility
- requirements.txt   – Python dependencies

##### Extract Key Architecture Insights from the README
- Two main runtime components:
1. Interpretable decision-tree “actor” (sympol.py, sdt.py)
2. Neural “critic” network (mlp.py)
- Integration point: PPO algorithm (in ppo.py / ppo_gymnax.py)
- Optional: dynamic rollout buffer, Optuna hyperparameter optimization, WandB tracking

##### Main Components
a. Command-Line Layer
– args.py: Parses flags (–env_id, –actor, –dynamic_buffer, –use_best_config, –optimize_config, –track, etc.)
b. Configuration Layer
– configs.py: Loads fixed or Optuna-suggested hyperparameters
c. Environment Adapter
– Gymnasium interface (ppo.py) or Gymnax vectorized interface (ppo_gymnax.py)
d. Rollout Buffer
– Dynamic buffer module within ppo[*].py (collects transitions for on-policy updates)
e. Actor Module
– sympol.py: Axis-aligned decision-tree policy with differentiable parameters
– sdt.py: Baseline soft/discretized tree policies
f. Critic Module
– mlp.py: Neural network approximating value function
g. PPO Trainer
– Core training loop: sample from env → store in buffer → compute actor & critic losses → backprop → update parameters
h. Logging & Experiment Management
– WandB (activated via –track)
– commands_to_run.txt and run_ppo.sh for batch experiments
i. Hyperparameter Optimization
– Optuna integrated in ppo.py when –optimize_config is passed
– configs.py.suggest_config() defines search spaces
– hpopt.db for trial persistence

##### Relationships and Data Flows
- Start: User invokes ppo.py or ppo_gymnax.py via run_ppo.sh or CLI
↓
- args.py reads flags → selects actor type (sympol vs sdt vs mlp actor) and other options
↓
- configs.py loads default or Optuna-suggested hyperparameters
↓
- Environment adapter creates vectorized or single-instance env
↓
- PPO trainer uses Rollout Buffer to gather (state, action, reward, next_state)
↓
- Actor (sympol.py) and Critic (mlp.py) compute losses on buffered data
↓
- Backpropagation & parameter update via JAX autodiff
↓
- Optional: logging to WandB, Graphviz export of tree structure
↓
- If –optimize_config: results passed back to Optuna, next trial begins

##### Highlighted Architectural Patterns and Principles
- Modular separation:
– CLI / Config / Environment / Policy / Value / Trainer / Logging
- Actor-Critic pattern: clear split of responsibilities
- Pluggable actors: can swap in Sympol (interpretable) or existing baselines (sdt.py)
- Extensibility for new environments (Gym vs Gymnax)
- Experiment reproducibility: fixed seeds, commands_to_run.txt, run_ppo.sh

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    subgraph "CLI Layer"
        CLI_ARGS["args.py"]:::code
    end

    %% Configuration Layer
    subgraph "Configuration Layer"
        CONFIG["configs.py"]:::code
    end

    %% Utilities
    subgraph "Shared Utilities"
        UTILS["utils.py"]:::code
    end

    %% Environment & Trainer Layer
    subgraph "Environment & Trainer"
        PPO["ppo.py"]:::code
        PPOX["ppo_gymnax.py"]:::code
    end

    %% Core Training Layer
    subgraph "Core Training Loop"
        BUFFER(("Rollout Buffer")):::datastore
        subgraph "Actor Modules"
            SYMPOL["sympol.py"]:::code
            SDT["sdt.py"]:::code
        end
        CRITIC["mlp.py"]:::code
    end

    %% External Services
    subgraph "External Services"
        GYM(["Gymnasium Env"]):::external
        GYMNX(["Gymnax Env"]):::external
        OPTUNA(["Optuna Trial DB"]):::external
        WANDB(["WandB Tracking"]):::external
    end

    %% Experiment Management
    subgraph "Experiment Management"
        RUNSH["run_ppo.sh"]:::code
        CMDS["commands_to_run.txt"]:::code
        REQS["requirements.txt"]:::code
        FIGS["figures/"]:::code
    end

    %% Data flows
    RUNSH -->|"invokes"| CLI_ARGS
    CLI_ARGS -->|"flags"| CONFIG
    CLI_ARGS -->|"flags"| PPO
    CLI_ARGS -->|"flags"| PPOX
    CONFIG -->|"hyperparams"| PPO
    CONFIG -->|"hyperparams"| PPOX
    UTILS -->|"setup logging, seeding"| CLI_ARGS
    UTILS -->|"helper funcs"| PPO
    UTILS -->|"helper funcs"| PPOX

    PPO -->|"creates env"| GYM
    PPOX -->|"creates env"| GYMNX

    GYM -->|"observations"| BUFFER
    GYMNX -->|"observations"| BUFFER

    BUFFER -->|"transitions"| SYMPOL
    BUFFER -->|"transitions"| SDT
    BUFFER -->|"transitions"| CRITIC

    SYMPOL -->|"actions, gradients"| PPO
    SDT -->|"actions, gradients"| PPO
    CRITIC -->|"value estimates"| PPO

    PPO -->|"updates params"| SYMPOL
    PPO -->|"updates params"| SDT
    PPO -->|"updates params"| CRITIC

    PPO -->|"log metrics"| WANDB
    PPOX -->|"log metrics"| WANDB

    PPO -->|"trial results"| OPTUNA
    CLI_ARGS -->|"--optimize_config"| OPTUNA

    RUNSH -->|"batch cmds"| CMDS
    PPO -->|"requires"| REQS
    PPOX -->|"requires"| REQS
    PPO -->|"outputs plots"| FIGS
    PPOX -->|"outputs plots"| FIGS

    %% Click Events
    click CLI_ARGS "https://github.com/s-marton/sympol/blob/master/args.py"
    click CONFIG "https://github.com/s-marton/sympol/blob/master/configs.py"
    click UTILS "https://github.com/s-marton/sympol/blob/master/utils.py"
    click PPO "https://github.com/s-marton/sympol/blob/master/ppo.py"
    click PPOX "https://github.com/s-marton/sympol/blob/master/ppo_gymnax.py"
    click SYMPOL "https://github.com/s-marton/sympol/blob/master/sympol.py"
    click SDT "https://github.com/s-marton/sympol/blob/master/sdt.py"
    click CRITIC "https://github.com/s-marton/sympol/blob/master/mlp.py"
    click RUNSH "https://github.com/s-marton/sympol/blob/master/run_ppo.sh"
    click CMDS "https://github.com/s-marton/sympol/blob/master/commands_to_run.txt"
    click REQS "https://github.com/s-marton/sympol/blob/master/requirements.txt"
    click FIGS "https://github.com/s-marton/sympol/tree/master/figures/"

    %% Styles
    classDef code fill:#D0E8FF,stroke:#0366D6,stroke-width:1px
    classDef datastore fill:#DCF5DC,stroke:#028A0F,stroke-width:1px
    classDef external fill:#FFE5B4,stroke:#CC7000,stroke-width:1px
```
