How daichikahtava/safe-rl works
This repository explores learning and enforcing temporal safety rules for a colour-bomb grid-world agent. The workflow samples labelled traces, learns or selects an LTLf rule, converts it to a safety DFA, and trains PPO behind a probabilistic shield; active learning also detects counterexamples for iterative repair. Separate capabilities include random, oracle-guided, and formula-constrained trace generation, baseline training, and evaluation/plotting. The sampled files establish the main training and sampling links; evaluation modules and some sampler variants are represented from the tree and index, so their internal connections are not inferred.
Main entry points: train_shield_passive.py Passive shield PPO · train_shield_active.py Active CEGIS PPO · train_shield_oracle.py Oracle shield PPO · sample_oracle.py Oracle sampling
Components 20
Grid World
- World v2
envs/colour_bomb_grid_world_v2.py - World v3
envs/colour_bomb_grid_world_v3.py - Rule monitors
wrappers.py
Trace Generation
- Trace utilities
sampler/sample_utils.py - Random sampling
sampler/sample_random.py - Oracle sampling
sampler/sample_oracle.py - Formula sampling
sampler/sample_formula_trunc_neg.py - Other samplers
sampler
Formal Safety
- Formula learner
bolt.py - LTLf parser and DFA
formula_parser.py
Training and Evaluation
- Baseline PPO
train/train_baseline.py - Passive shield PPO
train/train_shield_passive.py - Oracle shield PPO
train/train_shield_oracle.py - Active CEGIS PPO
train/train_shield_active.py - Probabilistic shield
- Evaluation and plots
eval - Near-miss analysis
near_misses.py
Other components
- Researcher
- Bolt engine
- LTLf2DFA
Connections 33
- Researcher → Random sampling: requests samples
- Researcher → Oracle sampling: requests samples
- Researcher → Formula sampling: requests samples
- Researcher → Baseline PPO: starts training
- Researcher → Passive shield PPO: starts training
- Researcher → Oracle shield PPO: starts training
- Researcher → Active CEGIS PPO: starts training
- Random sampling → World v3: samples rollouts
sampler/sample_random.py - Random sampling → Trace utilities: evaluates and saves
sampler/sample_random.py - Oracle sampling → World v3: samples rollouts
sampler/sample_oracle.py - Oracle sampling → LTLf parser and DFA: builds shield DFA
sampler/sample_oracle.py - Oracle sampling → Oracle shield PPO: loads rule formulas
sampler/sample_oracle.py - Oracle sampling → Trace utilities: evaluates and saves
sampler/sample_oracle.py - Formula sampling → Trace utilities: uses trace rules
sampler/sample_formula_trunc_neg.py - Formula learner → Bolt engine: runs synthesis
bolt.py - LTLf parser and DFA → LTLf2DFA: converts formula
formula_parser.py - Baseline PPO → World v2: creates environment
train/train_baseline.py - Baseline PPO → Rule monitors: uses wrappers
train/train_baseline.py - Passive shield PPO → Formula learner: learns formula
train/train_shield_passive.py - Passive shield PPO → LTLf parser and DFA: builds safety DFA
train/train_shield_passive.py - Passive shield PPO → World v2: creates environment
train/train_shield_passive.py - Passive shield PPO → Probabilistic shield: wraps environment
train/train_shield_passive.py - Passive shield PPO → Rule monitors: tracks violations
train/train_shield_passive.py - Oracle shield PPO → LTLf parser and DFA: builds safety DFA
train/train_shield_oracle.py - Oracle shield PPO → World v2: creates environment
train/train_shield_oracle.py - Oracle shield PPO → Probabilistic shield: wraps environments
train/train_shield_oracle.py - Oracle shield PPO → Rule monitors: tracks violations
train/train_shield_oracle.py - Active CEGIS PPO → Formula learner: learns formula
train/train_shield_active.py - Active CEGIS PPO → LTLf parser and DFA: builds safety DFA
train/train_shield_active.py - Active CEGIS PPO → World v3: creates environment
train/train_shield_active.py - Active CEGIS PPO → Probabilistic shield: wraps environment
train/train_shield_active.py - Active CEGIS PPO → Random sampling: uses random sampler
train/train_shield_active.py - Evaluation and plots → Researcher: reports results