# daichikahtava/safe-rl architecture

> Architecture diagram of the GitHub repository daichikahtava/safe-rl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/daichikahtava/safe-rl
- Repository: https://github.com/daichikahtava/safe-rl

## Overview

This repository explores learning and enforcing temporal safety rules for a colour-bomb grid-world agent. The workflow samples labelled traces, learns or selects an LTLf rule, converts it to a safety DFA, and trains PPO behind a probabilistic shield; active learning also detects counterexamples for iterative repair. Separate capabilities include random, oracle-guided, and formula-constrained trace generation, baseline training, and evaluation/plotting. The sampled files establish the main training and sampling links; evaluation modules and some sampler variants are represented from the tree and index, so their internal connections are not inferred.

Main entry points: [`train/train_shield_passive.py`](https://github.com/daichikahtava/safe-rl/blob/main/train/train_shield_passive.py) (Passive shield PPO), [`train/train_shield_active.py`](https://github.com/daichikahtava/safe-rl/blob/main/train/train_shield_active.py) (Active CEGIS PPO), [`train/train_shield_oracle.py`](https://github.com/daichikahtava/safe-rl/blob/main/train/train_shield_oracle.py) (Oracle shield PPO), [`sampler/sample_oracle.py`](https://github.com/daichikahtava/safe-rl/blob/main/sampler/sample_oracle.py) (Oracle sampling)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_world["Grid World"]
  node_world_v2["World v2"]
  node_world_v3["World v3"]
  node_rule_wrappers["Rule monitors<br/>[wrappers.py]"]
end

subgraph group_traces["Trace Generation"]
  node_trace_utils["Trace utilities<br/>[sample_utils.py]"]
  node_random_sampling["Random sampling<br/>[sample_random.py]"]
  node_oracle_sampling["Oracle sampling<br/>[sample_oracle.py]"]
  node_formula_sampling["Formula sampling"]
  node_other_sampling["Other samplers"]
end

subgraph group_formal["Formal Safety"]
  node_bolt["Formula learner<br/>[bolt.py]"]
  node_formula_parser["LTLf parser and DFA<br/>[formula_parser.py]"]
end

subgraph group_learning["Training and Evaluation"]
  node_baseline["Baseline PPO<br/>[train_baseline.py]"]
  node_passive_training["Passive shield PPO"]
  node_oracle_training["Oracle shield PPO"]
  node_active_training["Active CEGIS PPO"]
  node_prob_shield["Probabilistic shield"]
  node_evaluation["Evaluation and plots"]
  node_near_misses["Near-miss analysis<br/>[near_misses.py]"]
end

node_operator(("Researcher"))
node_bolt_engine["Bolt engine"]
node_ltlf_tool["LTLf2DFA"]

node_operator -->|"requests samples"| node_random_sampling
node_operator -->|"requests samples"| node_oracle_sampling
node_operator -->|"requests samples"| node_formula_sampling
node_operator -->|"starts training"| node_baseline
node_operator -->|"starts training"| node_passive_training
node_operator -->|"starts training"| node_oracle_training
node_operator -->|"starts training"| node_active_training
node_random_sampling -->|"samples rollouts"| node_world_v3
node_random_sampling -->|"evaluates and saves"| node_trace_utils
node_oracle_sampling -->|"samples rollouts"| node_world_v3
node_oracle_sampling -->|"builds shield DFA"| node_formula_parser
node_oracle_sampling -->|"loads rule formulas"| node_oracle_training
node_oracle_sampling -->|"evaluates and saves"| node_trace_utils
node_formula_sampling -->|"uses trace rules"| node_trace_utils
node_bolt -->|"runs synthesis"| node_bolt_engine
node_formula_parser -->|"converts formula"| node_ltlf_tool
node_baseline -->|"creates environment"| node_world_v2
node_baseline -->|"uses wrappers"| node_rule_wrappers
node_passive_training -->|"learns formula"| node_bolt
node_passive_training -->|"builds safety DFA"| node_formula_parser
node_passive_training -->|"creates environment"| node_world_v2
node_passive_training -->|"wraps environment"| node_prob_shield
node_passive_training -->|"tracks violations"| node_rule_wrappers
node_oracle_training -->|"builds safety DFA"| node_formula_parser
node_oracle_training -->|"creates environment"| node_world_v2
node_oracle_training -->|"wraps environments"| node_prob_shield
node_oracle_training -->|"tracks violations"| node_rule_wrappers
node_active_training -->|"learns formula"| node_bolt
node_active_training -->|"builds safety DFA"| node_formula_parser
node_active_training -->|"creates environment"| node_world_v3
node_active_training -->|"wraps environment"| node_prob_shield
node_active_training -->|"uses random sampler"| node_random_sampling
node_evaluation -.->|"reports results"| node_operator

click node_world_v2 "https://github.com/daichikahtava/safe-rl/blob/main/envs/colour_bomb_grid_world_v2.py"
click node_world_v3 "https://github.com/daichikahtava/safe-rl/blob/main/envs/colour_bomb_grid_world_v3.py"
click node_rule_wrappers "https://github.com/daichikahtava/safe-rl/blob/main/wrappers.py"
click node_trace_utils "https://github.com/daichikahtava/safe-rl/blob/main/sampler/sample_utils.py"
click node_random_sampling "https://github.com/daichikahtava/safe-rl/blob/main/sampler/sample_random.py"
click node_oracle_sampling "https://github.com/daichikahtava/safe-rl/blob/main/sampler/sample_oracle.py"
click node_formula_sampling "https://github.com/daichikahtava/safe-rl/blob/main/sampler/sample_formula_trunc_neg.py"
click node_other_sampling "https://github.com/daichikahtava/safe-rl/tree/main/sampler"
click node_bolt "https://github.com/daichikahtava/safe-rl/blob/main/bolt.py"
click node_formula_parser "https://github.com/daichikahtava/safe-rl/blob/main/formula_parser.py"
click node_baseline "https://github.com/daichikahtava/safe-rl/blob/main/train/train_baseline.py"
click node_passive_training "https://github.com/daichikahtava/safe-rl/blob/main/train/train_shield_passive.py"
click node_oracle_training "https://github.com/daichikahtava/safe-rl/blob/main/train/train_shield_oracle.py"
click node_active_training "https://github.com/daichikahtava/safe-rl/blob/main/train/train_shield_active.py"
click node_evaluation "https://github.com/daichikahtava/safe-rl/tree/main/eval"
click node_near_misses "https://github.com/daichikahtava/safe-rl/blob/main/near_misses.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_world_v2,node_world_v3,node_rule_wrappers toneBlue
class node_trace_utils,node_random_sampling,node_oracle_sampling,node_formula_sampling,node_other_sampling toneAmber
class node_bolt,node_formula_parser toneMint
class node_baseline,node_passive_training,node_oracle_training,node_active_training,node_prob_shield,node_evaluation,node_near_misses toneRose
class node_operator,node_bolt_engine,node_ltlf_tool toneIndigo
```

## Components

### Grid World

- [World v2](https://github.com/daichikahtava/safe-rl/blob/main/envs/colour_bomb_grid_world_v2.py): `envs/colour_bomb_grid_world_v2.py`
- [World v3](https://github.com/daichikahtava/safe-rl/blob/main/envs/colour_bomb_grid_world_v3.py): `envs/colour_bomb_grid_world_v3.py`
- [Rule monitors](https://github.com/daichikahtava/safe-rl/blob/main/wrappers.py): `wrappers.py`

### Trace Generation

- [Trace utilities](https://github.com/daichikahtava/safe-rl/blob/main/sampler/sample_utils.py): `sampler/sample_utils.py`
- [Random sampling](https://github.com/daichikahtava/safe-rl/blob/main/sampler/sample_random.py): `sampler/sample_random.py`
- [Oracle sampling](https://github.com/daichikahtava/safe-rl/blob/main/sampler/sample_oracle.py): `sampler/sample_oracle.py`
- [Formula sampling](https://github.com/daichikahtava/safe-rl/blob/main/sampler/sample_formula_trunc_neg.py): `sampler/sample_formula_trunc_neg.py`
- [Other samplers](https://github.com/daichikahtava/safe-rl/tree/main/sampler): `sampler`

### Formal Safety

- [Formula learner](https://github.com/daichikahtava/safe-rl/blob/main/bolt.py): `bolt.py`
- [LTLf parser and DFA](https://github.com/daichikahtava/safe-rl/blob/main/formula_parser.py): `formula_parser.py`

### Training and Evaluation

- [Baseline PPO](https://github.com/daichikahtava/safe-rl/blob/main/train/train_baseline.py): `train/train_baseline.py`
- [Passive shield PPO](https://github.com/daichikahtava/safe-rl/blob/main/train/train_shield_passive.py): `train/train_shield_passive.py`
- [Oracle shield PPO](https://github.com/daichikahtava/safe-rl/blob/main/train/train_shield_oracle.py): `train/train_shield_oracle.py`
- [Active CEGIS PPO](https://github.com/daichikahtava/safe-rl/blob/main/train/train_shield_active.py): `train/train_shield_active.py`
- **Probabilistic shield**
- [Evaluation and plots](https://github.com/daichikahtava/safe-rl/tree/main/eval): `eval`
- [Near-miss analysis](https://github.com/daichikahtava/safe-rl/blob/main/near_misses.py): `near_misses.py`

### Other components

- **Researcher**
- **Bolt engine**
- **LTLf2DFA**

## Connections

- Researcher → Random sampling: requests samples
- Researcher → Oracle sampling: requests samples
- Researcher → Formula sampling: requests samples
- Researcher → Baseline PPO: starts training
- Researcher → Passive shield PPO: starts training
- Researcher → Oracle shield PPO: starts training
- Researcher → Active CEGIS PPO: starts training
- Random sampling → World v3: samples rollouts (evidence: `sampler/sample_random.py`)
- Random sampling → Trace utilities: evaluates and saves (evidence: `sampler/sample_random.py`)
- Oracle sampling → World v3: samples rollouts (evidence: `sampler/sample_oracle.py`)
- Oracle sampling → LTLf parser and DFA: builds shield DFA (evidence: `sampler/sample_oracle.py`)
- Oracle sampling → Oracle shield PPO: loads rule formulas (evidence: `sampler/sample_oracle.py`)
- Oracle sampling → Trace utilities: evaluates and saves (evidence: `sampler/sample_oracle.py`)
- Formula sampling → Trace utilities: uses trace rules (evidence: `sampler/sample_formula_trunc_neg.py`)
- Formula learner → Bolt engine: runs synthesis (evidence: `bolt.py`)
- LTLf parser and DFA → LTLf2DFA: converts formula (evidence: `formula_parser.py`)
- Baseline PPO → World v2: creates environment (evidence: `train/train_baseline.py`)
- Baseline PPO → Rule monitors: uses wrappers (evidence: `train/train_baseline.py`)
- Passive shield PPO → Formula learner: learns formula (evidence: `train/train_shield_passive.py`)
- Passive shield PPO → LTLf parser and DFA: builds safety DFA (evidence: `train/train_shield_passive.py`)
- Passive shield PPO → World v2: creates environment (evidence: `train/train_shield_passive.py`)
- Passive shield PPO → Probabilistic shield: wraps environment (evidence: `train/train_shield_passive.py`)
- Passive shield PPO → Rule monitors: tracks violations (evidence: `train/train_shield_passive.py`)
- Oracle shield PPO → LTLf parser and DFA: builds safety DFA (evidence: `train/train_shield_oracle.py`)
- Oracle shield PPO → World v2: creates environment (evidence: `train/train_shield_oracle.py`)
- Oracle shield PPO → Probabilistic shield: wraps environments (evidence: `train/train_shield_oracle.py`)
- Oracle shield PPO → Rule monitors: tracks violations (evidence: `train/train_shield_oracle.py`)
- Active CEGIS PPO → Formula learner: learns formula (evidence: `train/train_shield_active.py`)
- Active CEGIS PPO → LTLf parser and DFA: builds safety DFA (evidence: `train/train_shield_active.py`)
- Active CEGIS PPO → World v3: creates environment (evidence: `train/train_shield_active.py`)
- Active CEGIS PPO → Probabilistic shield: wraps environment (evidence: `train/train_shield_active.py`)
- Active CEGIS PPO → Random sampling: uses random sampler (evidence: `train/train_shield_active.py`)
- Evaluation and plots → Researcher: reports results