How sohanpanda104/rl_vision_based_interception works
1. This repository is a three-stage reinforcement learning pipeline for a simulated Franka Panda arm doing pick-and-place on a moving conveyor object. The architecture is centered around stage-specific training scripts, with each stage owning its own environment setup, policy choice, checkpoints, and evaluation outputs.
2. The main runtime stack is Python with PyBullet and panda-gym for simulation, and Stable-Baselines3 for training. The project uses PPO for Stages 1 and 2, then switches to RecurrentPPO with an LSTM policy in Stage 3 to handle partial observability when the object is hidden.
3. The top-level execution boundary is the shell entrypoint run.sh, which likely sequences the stage scripts. The stage scripts are stage1/stage1.py, stage2/sim3d_stage2.py, stage2/test_sim3d_stage2.py, stage2/plot_training_metrics.py, and stage3/stage3.py.
4. Stage 1 is the simplest environment: the object moves at constant velocity, and the policy learns interception, grasping, and lifting. Its outputs live under stage1/models_3d_s2/, including periodic model checkpoints, a final model, normalization state, and a generated demo video.
5. Stage 2 adds domain randomization. The environment varies belt speed, object mass, and lateral drift per episode, forcing the policy to generalize rather than memorize a single trajectory. This stage has a clearer experiment-management layout: checkpoints, a best/ model, an eval/ directory with evaluations.npz, plots, and test videos.
6. Stage 3 introduces the main architectural challenge: occlusion by a tunnel. The observation becomes partially observable, so the policy must carry temporal state across steps. That is why Stage 3 uses a recurrent policy and stores its own recurrent checkpoints plus vector normalization state.
7. Each stage uses a dense reward design with shared structure but increasing difficulty. The reward flow is from environment state to shaped reward terms such as approach, hover, lift, target, and success; Stage 3 adds an occlusion-specific penalty to discourage premature catches before the tunnel.
8. A key data boundary in the repo is the observation/preprocessing layer. The policy receives normalized inputs such as end-effector state, block position/velocity when visible, and stage-specific context like belt speed, mass, and occlusion flags. The vecnormalize.pkl files indicate that observation/reward normalization is persisted separately from the policy weights.
9. A second major boundary is artifact management. Model checkpoints are saved as .zip files throughout training, while the evaluation pipeline emits .npz summaries and rendered .mp4 videos. This makes the repo both a training codebase and an experiment archive.
10. The stage2 plotting script sits outside the core training loop and consumes evaluation/training outputs to produce metrics visualizations. That makes the repo’s tooling pipeline roughly: train in simulation, periodically checkpoint, evaluate into structured files, then post-process into plots and videos.
11. There is no server, database, or external API layer here; the architecture is mostly offline simulation and training. The important operational concerns are reproducibility, checkpoint selection, normalization persistence, and stage-to-stage curriculum progression.
12. If you are drawing the architecture, the clearest diagram is: stage-specific environment script on the left, PyBullet/panda-gym simulation in the middle, Stable-Baselines3 policy training on the right, and then separate artifact sinks for checkpoints, normalization state, evaluation metrics, plots, and videos.
Main entry points: stage1.py Stage 1 train · sim3d_stage2.py Stage 2 train · stage3.py Stage 3 train · models_3d_s2 Stage 2 models
Components 18
Orchestration
Top-level entrypoint and stage sequencing
- run.sh · Launches the staged training workflow
run.sh
Stage 1
Constant-speed interception and lift training
- Stage 1 train · Trains the simplest pick-and-place policy
stage1/stage1.py
Stage 2
Domain-randomized generalization and evaluation
- Stage 2 train · Runs randomized conveyor training
stage2/sim3d_stage2.py - Stage 2 test · Evaluates the stage 2 policy and records test video
stage2/test_sim3d_stage2.py - Stage 2 plots · Turns training and evaluation outputs into plots
stage2/plot_training_metrics.py
Stage 3
Occlusion-aware recurrent training
- Stage 3 train · Trains the occlusion-aware recurrent policy
stage3/stage3.py
Shared Runtime
Simulation, learning, and preprocessing layers
- PyBullet sim · Robot and conveyor physics for the task
- panda-gym · Franka Panda environment wrapper and task structure
- SB3 · Training and evaluation framework used across stages
- PPO · Feedforward policy used in stages 1 and 2
- RecurrentPPO · LSTM-based policy used in stage 3
- VecNormalize · Persists observation and reward normalization state
- Obs + reward · Normalizes observations and shapes rewards for each stage
Artifacts
Checkpoints, normalization state, metrics, and media outputs
- Stage 1 models · Stage 1 zip checkpoints, final model, and demo video
stage1/models_3d_s2 - Stage 2 models · Stage 2 checkpoints, best model, eval files, plots, and test videos
stage2/models_3d_s2 - Stage 3 models · Stage 3 checkpoints, best model, eval files, and normalization state
stage3/models_3d_s3 - Eval metrics · Structured evaluation results consumed by plotting and analysis
- Videos + plots · Generated MP4 demos, test clips, and metric plots
Connections 25
- run.sh → Stage 1 train: stage 1 · Starts the first curriculum stage
- run.sh → Stage 2 train: stage 2 · Launches randomized training
- run.sh → Stage 3 train: stage 3 · Launches recurrent training
- Stage 1 train → panda-gym: env setup · Builds the task environment
- Stage 2 train → panda-gym: env setup · Builds the randomized task environment
- Stage 3 train → panda-gym: env setup · Builds the partially observable task environment
- panda-gym → PyBullet sim: backed by · Uses PyBullet for simulation
- Stage 1 train → SB3: train · Trains with Stable-Baselines3
- Stage 2 train → SB3: train · Trains with Stable-Baselines3
- Stage 3 train → SB3: train · Trains with Stable-Baselines3
- SB3 → PPO: stages 1-2 · Uses PPO for feedforward policies
- SB3 → RecurrentPPO: stage 3 · Uses recurrent PPO with LSTM policy
- Stage 1 train → Obs + reward: reward shaping · Computes shaped rewards and normalized observations
- Stage 2 train → Obs + reward: reward shaping · Applies domain-randomized reward/observation processing
- Stage 3 train → Obs + reward: reward shaping · Adds occlusion-aware shaping and hidden-state cues
- Obs + reward → VecNormalize: persisted by · Stores normalization separately from policy weights
- Stage 1 train → Stage 1 models: writes · Saves stage 1 checkpoints and final model
- Stage 2 train → Stage 2 models: writes · Saves stage 2 checkpoints and artifacts
- Stage 2 test → Stage 2 models: reads/writes · Consumes trained models and emits evaluation outputs
- Stage 2 plots → Stage 2 models: consumes · Reads eval data and training logs for plotting
- Stage 3 train → Stage 3 models: writes · Saves stage 3 checkpoints and normalization state
- Stage 2 models → Eval metrics: contains · Holds evaluations.npz and related summaries
- Stage 3 models → Eval metrics: contains · Holds evaluations.npz and related summaries
- Stage 1 models → Videos + plots: contains · Holds demo video output
- Stage 2 models → Videos + plots: contains · Holds test videos and plots