How s-marton/sympol works
Project Type and Purpose
• Type: A research-grade on-policy actor-critic reinforcement-learning framework, specialized in learning symbolic decision-tree policies (SYMPOL) with PPO.
• Purpose: Provide an end-to-end pipeline for training interpretable tree-based actors alongside neural critics, with optional hyperparameter optimization and logging.
File Structure
Top-level modules:
• args.py – command-line interface and argument parsing
• configs.py – hyperparameter definitions and Optuna search space
• utils.py – shared helper functions (seeding, logging setup)
• sympol.py – implementation of the symbolic decision-tree actor
• sdt.py – discretized/soft decision-tree baselines
• mlp.py – multi-layer perceptron critic
• ppo.py – main training script for Gymnasium environments
• ppo_gymnax.py – main training script for vectorized Gymnax environments
• run_ppo.sh – shell helper for parallel hyperparameter studies
• commands_to_run.txt– exact experiment commands for reproducibility
• requirements.txt – Python dependencies
Extract Key Architecture Insights from the README
• Two main runtime components:
1. Interpretable decision-tree “actor” (sympol.py, sdt.py)
2. Neural “critic” network (mlp.py)
• Integration point: PPO algorithm (in ppo.py / ppo_gymnax.py)
• Optional: dynamic rollout buffer, Optuna hyperparameter optimization, WandB tracking
Main Components
a. Command-Line Layer
– args.py: Parses flags (–env_id, –actor, –dynamic_buffer, –use_best_config, –optimize_config, –track, etc.)
b. Configuration Layer
– configs.py: Loads fixed or Optuna-suggested hyperparameters
c. Environment Adapter
– Gymnasium interface (ppo.py) or Gymnax vectorized interface (ppo_gymnax.py)
d. Rollout Buffer
– Dynamic buffer module within ppo[*].py (collects transitions for on-policy updates)
e. Actor Module
– sympol.py: Axis-aligned decision-tree policy with differentiable parameters
– sdt.py: Baseline soft/discretized tree policies
f. Critic Module
– mlp.py: Neural network approximating value function
g. PPO Trainer
– Core training loop: sample from env → store in buffer → compute actor & critic losses → backprop → update parameters
h. Logging & Experiment Management
– WandB (activated via –track)
– commands_to_run.txt and run_ppo.sh for batch experiments
i. Hyperparameter Optimization
– Optuna integrated in ppo.py when –optimize_config is passed
– configs.py.suggest_config() defines search spaces
– hpopt.db for trial persistence
Relationships and Data Flows
• Start: User invokes ppo.py or ppo_gymnax.py via run_ppo.sh or CLI
↓
• args.py reads flags → selects actor type (sympol vs sdt vs mlp actor) and other options
↓
• configs.py loads default or Optuna-suggested hyperparameters
↓
• Environment adapter creates vectorized or single-instance env
↓
• PPO trainer uses Rollout Buffer to gather (state, action, reward, next_state)
↓
• Actor (sympol.py) and Critic (mlp.py) compute losses on buffered data
↓
• Backpropagation & parameter update via JAX autodiff
↓
• Optional: logging to WandB, Graphviz export of tree structure
↓
• If –optimize_config: results passed back to Optuna, next trial begins
Highlighted Architectural Patterns and Principles
• Modular separation:
– CLI / Config / Environment / Policy / Value / Trainer / Logging
• Actor-Critic pattern: clear split of responsibilities
• Pluggable actors: can swap in Sympol (interpretable) or existing baselines (sdt.py)
• Extensibility for new environments (Gym vs Gymnax)
• Experiment reproducibility: fixed seeds, commands_to_run.txt, run_ppo.sh