How nm512/dreamerv3-torch works
Project Overview
• Type: Reinforcement Learning research codebase (PyTorch) implementing the DreamerV3 algorithm.
• Purpose: Train a world-model-based RL agent across diverse environments (DMC, Atari, Crafter, Minecraft, etc.) with fixed hyperparameters.
Main Components
a. Configuration & Setup
• configs.yaml – holds experiment hyperparameters and environment specs
• Dockerfile & envs/setup_scripts – reproducible environment setup for Docker and shell
b. Environment Layer
• envs/ – wrappers and launch scripts for various environments (atari.py, dmc.py, minecraft.py, etc.)
• wrappers.py – generic observation/action normalization, frame stacking, reward clipping
c. Data Generation & Parallelization
• parallel.py – multi-process or multi-threaded rollout collector that runs envs in parallel
• exploration.py – exploration routines (e.g., ε-greedy or intrinsic rewards)
d. World Model & Agent
• networks.py – neural network definitions for representation (encoder/decoder), transition, reward, termination heads
• models.py – high-level world-model class that ties networks into a trainable unit (e.g., RSSM, world-step updates)
• tools.py – replay buffer, data preprocessing, sequence slicing, batch sampling
e. Training Orchestration
• dreamer.py – main entry point: parses configs, launches parallel rollouts, trains world model & actor-critic policy in an interleaved loop, logs metrics
f. Monitoring & Logging
• tensorboard (via logdir) – monitors losses, returns, value estimates, latent predictions
• imgs/ – stores example learning curves embedded in README
• On the left, show “Environment Pool”: multiple env instances spun up by parallel.py invoking scripts from envs/setup_scripts via wrappers.py.
• From Replay Buffer → World Model module: networks.py + models.py. Label: “sample batches.”
• At the bottom, external dependencies: DeepMind Control Suite, Atari-Py, MinerL, Crafter, Docker. Connect to Environment Pool.
Relationships & Dataflow
• Config → dreamer.py, parallel.py (initialization)
• dreamer.py → parallel.py → envs wrappers → Environments
• Environments → Replay Buffer
• Replay Buffer → World Model → Policy
• Policy → Environments (actions)
• dreamer.py → tensorboard (metrics), disk (checkpoints)
Architectural Patterns & Principles
• Modular separation:
• “Env layer” vs. “Model & Agent” vs. “Orchestration” vs. “Logging”
• Data-parallel rollouts and sequential training loops
• Use of a world model (latent dynamics) and imagination-based policy learning
• Configuration-driven experimentation (configs.yaml)