# aungkaung1928/microduck-rl architecture

> Architecture diagram of the GitHub repository aungkaung1928/microduck-rl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aungkaung1928/microduck-rl
- Repository: https://github.com/aungkaung1928/microduck-rl

## Overview

microduck-rl trains a CPU MuJoCo balance-and-push-recovery policy, evaluates it on nominal and held-out physics, and exports a verified ONNX policy. The workflow runs from the environment contract and PD baseline through parallel rollouts, PPO, domain-randomisation experiments, common episode metrics, evaluation, and export. The README documents these project stages; sampled source shows environment construction, worker behavior, randomisation, metrics, policy loading, and ONNX conversion. Some orchestration links rely on README documentation because the relevant source excerpts are partial.

Main entry points: [`ppo.py`](https://github.com/aungkaung1928/microduck-rl/blob/main/ppo.py) (PPO Trainer), [`export_onnx.py`](https://github.com/aungkaung1928/microduck-rl/blob/main/export_onnx.py) (ONNX Export), [`env.py`](https://github.com/aungkaung1928/microduck-rl/blob/main/env.py) (MuJoCo Environment), [`eval_policy.py`](https://github.com/aungkaung1928/microduck-rl/blob/main/eval_policy.py) (Policy Evaluation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_simulation["Simulation Contract"]
  node_common["Model Configuration<br/>[common.py]"]
  node_environment["MuJoCo Environment<br/>[env.py]"]
  node_baseline["PD Baseline<br/>[baseline.py]"]
  node_vecenv["Process Vector Env<br/>[vec_env.py]"]
  node_benchmark["CPU Throughput<br/>[bench.py]"]
end

subgraph group_learning["Policy Learning"]
  node_boxcheck["Training Guard<br/>[boxcheck.py]"]
  node_ppo["PPO Trainer<br/>[ppo.py]"]
  node_randomization["Domain Randomization<br/>[dr.py]"]
end

subgraph group_assessment["Evaluation"]
  node_metrics["Episode Metrics<br/>[metrics.py]"]
  node_evaluation["Policy Evaluation<br/>[eval_policy.py]"]
  node_gap["Held-out Gap Analysis<br/>[eval_gap.py]"]
end

subgraph group_deployment["Policy Export"]
  node_onnxexport["ONNX Export<br/>[export_onnx.py]"]
  node_onnxruntime["ONNX Runtime"]
  node_policyfile["ONNX Policy"]
end

node_user(("Experimenter"))

node_user -->|"measures"| node_benchmark
node_user -->|"measures"| node_baseline
node_common -->|"configures"| node_environment
node_environment -->|"simulates"| node_baseline
node_environment -->|"steps in workers"| node_vecenv
node_user -->|"starts training"| node_boxcheck
node_boxcheck -.->|"gates"| node_ppo
node_ppo -->|"collects rollouts"| node_vecenv
node_ppo -->|"trains with"| node_randomization
node_randomization -->|"changes reset physics"| node_environment
node_ppo -->|"measures episodes"| node_metrics
node_user -->|"evaluates policy"| node_evaluation
node_evaluation -->|"uses metrics"| node_metrics
node_evaluation -->|"loads policy"| node_ppo
node_gap -->|"compares variants"| node_evaluation
node_user -->|"exports policy"| node_onnxexport
node_onnxexport -->|"loads checkpoint"| node_ppo
node_onnxexport -->|"samples observations"| node_environment
node_onnxexport -->|"verifies and times"| node_onnxruntime
node_onnxexport -->|"writes"| node_policyfile

click node_common "https://github.com/aungkaung1928/microduck-rl/blob/main/common.py"
click node_environment "https://github.com/aungkaung1928/microduck-rl/blob/main/env.py"
click node_baseline "https://github.com/aungkaung1928/microduck-rl/blob/main/baseline.py"
click node_vecenv "https://github.com/aungkaung1928/microduck-rl/blob/main/vec_env.py"
click node_boxcheck "https://github.com/aungkaung1928/microduck-rl/blob/main/boxcheck.py"
click node_ppo "https://github.com/aungkaung1928/microduck-rl/blob/main/ppo.py"
click node_randomization "https://github.com/aungkaung1928/microduck-rl/blob/main/dr.py"
click node_metrics "https://github.com/aungkaung1928/microduck-rl/blob/main/metrics.py"
click node_evaluation "https://github.com/aungkaung1928/microduck-rl/blob/main/eval_policy.py"
click node_gap "https://github.com/aungkaung1928/microduck-rl/blob/main/eval_gap.py"
click node_onnxexport "https://github.com/aungkaung1928/microduck-rl/blob/main/export_onnx.py"
click node_benchmark "https://github.com/aungkaung1928/microduck-rl/blob/main/bench.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_common,node_environment,node_baseline,node_vecenv,node_benchmark toneBlue
class node_boxcheck,node_ppo,node_randomization toneAmber
class node_metrics,node_evaluation,node_gap toneMint
class node_onnxexport,node_onnxruntime,node_policyfile toneRose
class node_user toneIndigo
```

## Components

### Simulation Contract

- [Model Configuration](https://github.com/aungkaung1928/microduck-rl/blob/main/common.py): `common.py`
- [MuJoCo Environment](https://github.com/aungkaung1928/microduck-rl/blob/main/env.py): `env.py`
- [PD Baseline](https://github.com/aungkaung1928/microduck-rl/blob/main/baseline.py): `baseline.py`
- [Process Vector Env](https://github.com/aungkaung1928/microduck-rl/blob/main/vec_env.py): `vec_env.py`
- [CPU Throughput](https://github.com/aungkaung1928/microduck-rl/blob/main/bench.py): `bench.py`

### Policy Learning

- [Training Guard](https://github.com/aungkaung1928/microduck-rl/blob/main/boxcheck.py): `boxcheck.py`
- [PPO Trainer](https://github.com/aungkaung1928/microduck-rl/blob/main/ppo.py): `ppo.py`
- [Domain Randomization](https://github.com/aungkaung1928/microduck-rl/blob/main/dr.py): `dr.py`

### Evaluation

- [Episode Metrics](https://github.com/aungkaung1928/microduck-rl/blob/main/metrics.py): `metrics.py`
- [Policy Evaluation](https://github.com/aungkaung1928/microduck-rl/blob/main/eval_policy.py): `eval_policy.py`
- [Held-out Gap Analysis](https://github.com/aungkaung1928/microduck-rl/blob/main/eval_gap.py): `eval_gap.py`

### Policy Export

- [ONNX Export](https://github.com/aungkaung1928/microduck-rl/blob/main/export_onnx.py): `export_onnx.py`
- **ONNX Runtime**
- **ONNX Policy**

### Other components

- **Experimenter**

## Connections

- Experimenter → CPU Throughput: measures (evidence: `README.md`)
- Experimenter → PD Baseline: measures (evidence: `README.md`)
- Model Configuration → MuJoCo Environment: configures (evidence: `env.py`)
- MuJoCo Environment → PD Baseline: simulates (evidence: `baseline.py`)
- MuJoCo Environment → Process Vector Env: steps in workers (evidence: `vec_env.py`)
- Experimenter → Training Guard: starts training (evidence: `README.md`)
- Training Guard → PPO Trainer: gates (evidence: `ppo.py`)
- PPO Trainer → Process Vector Env: collects rollouts (evidence: `README.md`)
- PPO Trainer → Domain Randomization: trains with (evidence: `README.md`)
- Domain Randomization → MuJoCo Environment: changes reset physics (evidence: `env.py`)
- PPO Trainer → Episode Metrics: measures episodes (evidence: `README.md`)
- Experimenter → Policy Evaluation: evaluates policy (evidence: `README.md`)
- Policy Evaluation → Episode Metrics: uses metrics (evidence: `eval_policy.py`)
- Policy Evaluation → PPO Trainer: loads policy (evidence: `eval_policy.py`)
- Held-out Gap Analysis → Policy Evaluation: compares variants (evidence: `eval_gap.py`)
- Experimenter → ONNX Export: exports policy (evidence: `README.md`)
- ONNX Export → PPO Trainer: loads checkpoint (evidence: `export_onnx.py`)
- ONNX Export → MuJoCo Environment: samples observations (evidence: `export_onnx.py`)
- ONNX Export → ONNX Runtime: verifies and times (evidence: `export_onnx.py`)
- ONNX Export → ONNX Policy: writes (evidence: `README.md`)
