# sohanpanda104/rl_vision_based_interception architecture

> Architecture diagram of the GitHub repository sohanpanda104/rl_vision_based_interception, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sohanpanda104/rl_vision_based_interception
- Repository: https://github.com/sohanpanda104/rl_vision_based_interception

## Overview

1. This repository is a three-stage reinforcement learning pipeline for a simulated Franka Panda arm doing pick-and-place on a moving conveyor object. The architecture is centered around stage-specific training scripts, with each stage owning its own environment setup, policy choice, checkpoints, and evaluation outputs.

2. The main runtime stack is Python with PyBullet and panda-gym for simulation, and Stable-Baselines3 for training. The project uses PPO for Stages 1 and 2, then switches to RecurrentPPO with an LSTM policy in Stage 3 to handle partial observability when the object is hidden.

3. The top-level execution boundary is the shell entrypoint `run.sh`, which likely sequences the stage scripts. The stage scripts are `stage1/stage1.py`, `stage2/sim3d_stage2.py`, `stage2/test_sim3d_stage2.py`, `stage2/plot_training_metrics.py`, and `stage3/stage3.py`.

4. Stage 1 is the simplest environment: the object moves at constant velocity, and the policy learns interception, grasping, and lifting. Its outputs live under `stage1/models_3d_s2/`, including periodic model checkpoints, a final model, normalization state, and a generated demo video.

5. Stage 2 adds domain randomization. The environment varies belt speed, object mass, and lateral drift per episode, forcing the policy to generalize rather than memorize a single trajectory. This stage has a clearer experiment-management layout: checkpoints, a `best/` model, an `eval/` directory with `evaluations.npz`, plots, and test videos.

6. Stage 3 introduces the main architectural challenge: occlusion by a tunnel. The observation becomes partially observable, so the policy must carry temporal state across steps. That is why Stage 3 uses a recurrent policy and stores its own recurrent checkpoints plus vector normalization state.

7. Each stage uses a dense reward design with shared structure but increasing difficulty. The reward flow is from environment state to shaped reward terms such as approach, hover, lift, target, and success; Stage 3 adds an occlusion-specific penalty to discourage premature catches before the tunnel.

8. A key data boundary in the repo is the observation/preprocessing layer. The policy receives normalized inputs such as end-effector state, block position/velocity when visible, and stage-specific context like belt speed, mass, and occlusion flags. The `vecnormalize.pkl` files indicate that observation/reward normalization is persisted separately from the policy weights.

9. A second major boundary is artifact management. Model checkpoints are saved as `.zip` files throughout training, while the evaluation pipeline emits `.npz` summaries and rendered `.mp4` videos. This makes the repo both a training codebase and an experiment archive.

10. The stage2 plotting script sits outside the core training loop and consumes evaluation/training outputs to produce metrics visualizations. That makes the repo’s tooling pipeline roughly: train in simulation, periodically checkpoint, evaluate into structured files, then post-process into plots and videos.

11. There is no server, database, or external API layer here; the architecture is mostly offline simulation and training. The important operational concerns are reproducibility, checkpoint selection, normalization persistence, and stage-to-stage curriculum progression.

12. If you are drawing the architecture, the clearest diagram is: stage-specific environment script on the left, PyBullet/panda-gym simulation in the middle, Stable-Baselines3 policy training on the right, and then separate artifact sinks for checkpoints, normalization state, evaluation metrics, plots, and videos.

Main entry points: [`stage1/stage1.py`](https://github.com/sohanpanda104/rl_vision_based_interception/blob/main/stage1/stage1.py) (Stage 1 train), [`stage2/sim3d_stage2.py`](https://github.com/sohanpanda104/rl_vision_based_interception/blob/main/stage2/sim3d_stage2.py) (Stage 2 train), [`stage3/stage3.py`](https://github.com/sohanpanda104/rl_vision_based_interception/blob/main/stage3/stage3.py) (Stage 3 train), [`stage2/models_3d_s2`](https://github.com/sohanpanda104/rl_vision_based_interception/tree/main/stage2/models_3d_s2) (Stage 2 models)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_run_sh["run.sh<br/>shell entrypoint<br/>[run.sh]"]
end

subgraph group_stage1["Stage 1"]
  node_stage1_script["Stage 1 train<br/>training script<br/>[stage1.py]"]
end

subgraph group_stage2["Stage 2"]
  node_stage2_train_script["Stage 2 train<br/>training script<br/>[sim3d_stage2.py]"]
  node_stage2_test_script["Stage 2 test<br/>evaluation script"]
  node_stage2_plot_script["Stage 2 plots<br/>metrics script"]
end

subgraph group_stage3["Stage 3"]
  node_stage3_script["Stage 3 train<br/>recurrent training script<br/>[stage3.py]"]
end

subgraph group_shared["Shared Runtime"]
  node_sim_env[("PyBullet sim<br/>physics simulation")]
  node_panda_gym["panda-gym<br/>robot envs"]
  node_sb3["SB3<br/>rl framework"]
  node_ppo{{"PPO<br/>policy optimizer"}}
  node_recurrent_ppo{{"RecurrentPPO<br/>sequence optimizer"}}
  node_vecnormalize["VecNormalize<br/>normalization layer"]
  node_obs_reward["Obs + reward<br/>preprocessing"]
end

subgraph group_artifacts["Artifacts"]
  node_stage1_ckpts[("Stage 1 models<br/>checkpoints")]
  node_stage2_ckpts[("Stage 2 models<br/>experiment outputs")]
  node_stage3_ckpts[("Stage 3 models<br/>experiment outputs")]
  node_eval_metrics["Eval metrics<br/>summary files"]
  node_media_outputs["Videos + plots<br/>rendered outputs"]
end

node_run_sh -->|"stage 1"| node_stage1_script
node_run_sh -->|"stage 2"| node_stage2_train_script
node_run_sh -->|"stage 3"| node_stage3_script
node_stage1_script -->|"env setup"| node_panda_gym
node_stage2_train_script -->|"env setup"| node_panda_gym
node_stage3_script -->|"env setup"| node_panda_gym
node_panda_gym -->|"backed by"| node_sim_env
node_stage1_script -->|"train"| node_sb3
node_stage2_train_script -->|"train"| node_sb3
node_stage3_script -->|"train"| node_sb3
node_sb3 -->|"stages 1-2"| node_ppo
node_sb3 -->|"stage 3"| node_recurrent_ppo
node_stage1_script -->|"reward shaping"| node_obs_reward
node_stage2_train_script -->|"reward shaping"| node_obs_reward
node_stage3_script -->|"reward shaping"| node_obs_reward
node_obs_reward -->|"persisted by"| node_vecnormalize
node_stage1_script -->|"writes"| node_stage1_ckpts
node_stage2_train_script -->|"writes"| node_stage2_ckpts
node_stage2_test_script -->|"reads/writes"| node_stage2_ckpts
node_stage2_plot_script -->|"consumes"| node_stage2_ckpts
node_stage3_script -->|"writes"| node_stage3_ckpts
node_stage2_ckpts -->|"contains"| node_eval_metrics
node_stage3_ckpts -->|"contains"| node_eval_metrics
node_stage1_ckpts -->|"contains"| node_media_outputs
node_stage2_ckpts -->|"contains"| node_media_outputs

click node_run_sh "https://github.com/sohanpanda104/rl_vision_based_interception/blob/main/run.sh"
click node_stage1_script "https://github.com/sohanpanda104/rl_vision_based_interception/blob/main/stage1/stage1.py"
click node_stage2_train_script "https://github.com/sohanpanda104/rl_vision_based_interception/blob/main/stage2/sim3d_stage2.py"
click node_stage2_test_script "https://github.com/sohanpanda104/rl_vision_based_interception/blob/main/stage2/test_sim3d_stage2.py"
click node_stage2_plot_script "https://github.com/sohanpanda104/rl_vision_based_interception/blob/main/stage2/plot_training_metrics.py"
click node_stage3_script "https://github.com/sohanpanda104/rl_vision_based_interception/blob/main/stage3/stage3.py"
click node_stage1_ckpts "https://github.com/sohanpanda104/rl_vision_based_interception/tree/main/stage1/models_3d_s2"
click node_stage2_ckpts "https://github.com/sohanpanda104/rl_vision_based_interception/tree/main/stage2/models_3d_s2"
click node_stage3_ckpts "https://github.com/sohanpanda104/rl_vision_based_interception/tree/main/stage3/models_3d_s3"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_run_sh toneBlue
class node_stage1_script toneAmber
class node_stage2_train_script,node_stage2_test_script,node_stage2_plot_script toneMint
class node_stage3_script toneRose
class node_sim_env,node_panda_gym,node_sb3,node_ppo,node_recurrent_ppo,node_vecnormalize,node_obs_reward toneIndigo
class node_stage1_ckpts,node_stage2_ckpts,node_stage3_ckpts,node_eval_metrics,node_media_outputs toneTeal
```

## Components

### Orchestration

Top-level entrypoint and stage sequencing

- [run.sh](https://github.com/sohanpanda104/rl_vision_based_interception/blob/main/run.sh): shell entrypoint · Launches the staged training workflow · `run.sh`

### Stage 1

Constant-speed interception and lift training

- [Stage 1 train](https://github.com/sohanpanda104/rl_vision_based_interception/blob/main/stage1/stage1.py): training script · Trains the simplest pick-and-place policy · `stage1/stage1.py`

### Stage 2

Domain-randomized generalization and evaluation

- [Stage 2 train](https://github.com/sohanpanda104/rl_vision_based_interception/blob/main/stage2/sim3d_stage2.py): training script · Runs randomized conveyor training · `stage2/sim3d_stage2.py`
- [Stage 2 test](https://github.com/sohanpanda104/rl_vision_based_interception/blob/main/stage2/test_sim3d_stage2.py): evaluation script · Evaluates the stage 2 policy and records test video · `stage2/test_sim3d_stage2.py`
- [Stage 2 plots](https://github.com/sohanpanda104/rl_vision_based_interception/blob/main/stage2/plot_training_metrics.py): metrics script · Turns training and evaluation outputs into plots · `stage2/plot_training_metrics.py`

### Stage 3

Occlusion-aware recurrent training

- [Stage 3 train](https://github.com/sohanpanda104/rl_vision_based_interception/blob/main/stage3/stage3.py): recurrent training script · Trains the occlusion-aware recurrent policy · `stage3/stage3.py`

### Shared Runtime

Simulation, learning, and preprocessing layers

- **PyBullet sim**: physics simulation · Robot and conveyor physics for the task
- **panda-gym**: robot envs · Franka Panda environment wrapper and task structure
- **SB3**: rl framework · Training and evaluation framework used across stages
- **PPO**: policy optimizer · Feedforward policy used in stages 1 and 2
- **RecurrentPPO**: sequence optimizer · LSTM-based policy used in stage 3
- **VecNormalize**: normalization layer · Persists observation and reward normalization state
- **Obs + reward**: preprocessing · Normalizes observations and shapes rewards for each stage

### Artifacts

Checkpoints, normalization state, metrics, and media outputs

- [Stage 1 models](https://github.com/sohanpanda104/rl_vision_based_interception/tree/main/stage1/models_3d_s2): checkpoints · Stage 1 zip checkpoints, final model, and demo video · `stage1/models_3d_s2`
- [Stage 2 models](https://github.com/sohanpanda104/rl_vision_based_interception/tree/main/stage2/models_3d_s2): experiment outputs · Stage 2 checkpoints, best model, eval files, plots, and test videos · `stage2/models_3d_s2`
- [Stage 3 models](https://github.com/sohanpanda104/rl_vision_based_interception/tree/main/stage3/models_3d_s3): experiment outputs · Stage 3 checkpoints, best model, eval files, and normalization state · `stage3/models_3d_s3`
- **Eval metrics**: summary files · Structured evaluation results consumed by plotting and analysis
- **Videos + plots**: rendered outputs · Generated MP4 demos, test clips, and metric plots

## Connections

- run.sh → Stage 1 train: stage 1 · Starts the first curriculum stage
- run.sh → Stage 2 train: stage 2 · Launches randomized training
- run.sh → Stage 3 train: stage 3 · Launches recurrent training
- Stage 1 train → panda-gym: env setup · Builds the task environment
- Stage 2 train → panda-gym: env setup · Builds the randomized task environment
- Stage 3 train → panda-gym: env setup · Builds the partially observable task environment
- panda-gym → PyBullet sim: backed by · Uses PyBullet for simulation
- Stage 1 train → SB3: train · Trains with Stable-Baselines3
- Stage 2 train → SB3: train · Trains with Stable-Baselines3
- Stage 3 train → SB3: train · Trains with Stable-Baselines3
- SB3 → PPO: stages 1-2 · Uses PPO for feedforward policies
- SB3 → RecurrentPPO: stage 3 · Uses recurrent PPO with LSTM policy
- Stage 1 train → Obs + reward: reward shaping · Computes shaped rewards and normalized observations
- Stage 2 train → Obs + reward: reward shaping · Applies domain-randomized reward/observation processing
- Stage 3 train → Obs + reward: reward shaping · Adds occlusion-aware shaping and hidden-state cues
- Obs + reward → VecNormalize: persisted by · Stores normalization separately from policy weights
- Stage 1 train → Stage 1 models: writes · Saves stage 1 checkpoints and final model
- Stage 2 train → Stage 2 models: writes · Saves stage 2 checkpoints and artifacts
- Stage 2 test → Stage 2 models: reads/writes · Consumes trained models and emits evaluation outputs
- Stage 2 plots → Stage 2 models: consumes · Reads eval data and training logs for plotting
- Stage 3 train → Stage 3 models: writes · Saves stage 3 checkpoints and normalization state
- Stage 2 models → Eval metrics: contains · Holds evaluations.npz and related summaries
- Stage 3 models → Eval metrics: contains · Holds evaluations.npz and related summaries
- Stage 1 models → Videos + plots: contains · Holds demo video output
- Stage 2 models → Videos + plots: contains · Holds test videos and plots
