# nm512/dreamerv3-torch architecture

> Architecture diagram of the GitHub repository nm512/dreamerv3-torch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nm512/dreamerv3-torch
- Repository: https://github.com/nm512/dreamerv3-torch

## Overview

##### Project Overview
- Type: Reinforcement Learning research codebase (PyTorch) implementing the DreamerV3 algorithm.
- Purpose: Train a world-model-based RL agent across diverse environments (DMC, Atari, Crafter, Minecraft, etc.) with fixed hyperparameters.

##### Main Components
a. Configuration & Setup
- configs.yaml – holds experiment hyperparameters and environment specs
- Dockerfile & envs/setup_scripts – reproducible environment setup for Docker and shell
b. Environment Layer
- envs/ – wrappers and launch scripts for various environments (atari.py, dmc.py, minecraft.py, etc.)
- wrappers.py – generic observation/action normalization, frame stacking, reward clipping
c. Data Generation & Parallelization
- parallel.py – multi-process or multi-threaded rollout collector that runs envs in parallel
- exploration.py – exploration routines (e.g., ε-greedy or intrinsic rewards)
d. World Model & Agent
- networks.py – neural network definitions for representation (encoder/decoder), transition, reward, termination heads
- models.py – high-level world-model class that ties networks into a trainable unit (e.g., RSSM, world-step updates)
- tools.py – replay buffer, data preprocessing, sequence slicing, batch sampling
e. Training Orchestration
- dreamer.py – main entry point: parses configs, launches parallel rollouts, trains world model & actor-critic policy in an interleaved loop, logs metrics
f. Monitoring & Logging
- tensorboard (via logdir) – monitors losses, returns, value estimates, latent predictions
- imgs/ – stores example learning curves embedded in README

- On the left, show “Environment Pool”: multiple env instances spun up by parallel.py invoking scripts from envs/setup_scripts via wrappers.py.
- From Replay Buffer → World Model module: networks.py + models.py. Label: “sample batches.”
- At the bottom, external dependencies: DeepMind Control Suite, Atari-Py, MinerL, Crafter, Docker. Connect to Environment Pool.

##### Relationships & Dataflow
- Config → dreamer.py, parallel.py (initialization)
- dreamer.py → parallel.py → envs wrappers → Environments
- Environments → Replay Buffer
- Replay Buffer → World Model → Policy
- Policy → Environments (actions)
- dreamer.py → tensorboard (metrics), disk (checkpoints)

##### Architectural Patterns & Principles
- Modular separation:
- “Env layer” vs. “Model & Agent” vs. “Orchestration” vs. “Logging”
- Data-parallel rollouts and sequential training loops
- Use of a world model (latent dynamics) and imagination-based policy learning
- Configuration-driven experimentation (configs.yaml)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Configuration & Setup
    subgraph "Configuration & Setup"
        CFG["configs.yaml"]:::config
        DOCK["Dockerfile"]:::config
        SCRIPTS["envs/setup_scripts/"]:::config
    end

    %% Environment Layer
    subgraph "Environment Layer"
        WRAP["wrappers.py"]:::compute
        ENVS["envs/"]:::external
    end

    %% Data Generation & Parallelization
    subgraph "Data Generation & Parallelization"
        PAR["parallel.py"]:::compute
        EXP["exploration.py"]:::compute
        STORE((Replay Buffer)):::data
    end

    %% World Model & Agent
    subgraph "World Model & Agent"
        NET["networks.py"]:::compute
        MOD["models.py"]:::compute
        TOOLS["tools.py"]:::compute
        subgraph "World Model"
            REP["Representation (encoder/decoder)"]:::compute
            DYN["Dynamics (RSSM)"]:::compute
            HEAD["Heads (reward, discount)"]:::compute
        end
        subgraph "Policy / Actor-Critic"
            POL["Policy Module"]:::compute
        end
    end

    %% Training Orchestration
    subgraph "Training Orchestration"
        DREAM["dreamer.py"]:::compute
    end

    %% Monitoring & Logging
    subgraph "Monitoring & Logging"
        TB["TensorBoard"]:::external
        IMGS["imgs/"]:::data
    end

    %% External Dependencies
    subgraph "External Dependencies"
        DM["DeepMind Control Suite"]:::external
        AT["Atari-Py"]:::external
        CR["Crafter"]:::external
        ML["MinerL / Minecraft"]:::external
        DOCK_EXT["Docker"]:::external
    end

    %% Data Flows
    CFG -->|initializes| DREAM
    CFG -->|initializes| PAR
    DREAM -->|launches| PAR
    PAR -->|uses| WRAP
    WRAP -->|instantiates| ENVS
    ENVS -->|rollout trajectories| STORE
    STORE -->|sample batches| NET
    NET -->|parameters| MOD
    MOD -->|updates| TOOLS
    TOOLS -->|provides replay| STORE
    MOD -->|"imagined rollouts"| POL
    POL -->|actions| ENVS
    DREAM -->|orchestrates loop| STORE
    DREAM -->|orchestrates loop| NET
    DREAM -->|orchestrates loop| POL
    DREAM -->|logs metrics| TB
    DREAM -->|saves checkpoints| IMGS

    ENVS --> DM
    ENVS --> AT
    ENVS --> CR
    ENVS --> ML
    PAR --> DOCK_EXT

    %% Click Events
    click CFG "https://github.com/nm512/dreamerv3-torch/blob/main/configs.yaml"
    click DOCK "https://github.com/nm512/dreamerv3-torch/tree/main/Dockerfile"
    click SCRIPTS "https://github.com/nm512/dreamerv3-torch/tree/main/envs/setup_scripts/"
    click ENVS "https://github.com/nm512/dreamerv3-torch/tree/main/envs/"
    click WRAP "https://github.com/nm512/dreamerv3-torch/blob/main/envs/wrappers.py"
    click PAR "https://github.com/nm512/dreamerv3-torch/blob/main/parallel.py"
    click EXP "https://github.com/nm512/dreamerv3-torch/blob/main/exploration.py"
    click NET "https://github.com/nm512/dreamerv3-torch/blob/main/networks.py"
    click MOD "https://github.com/nm512/dreamerv3-torch/blob/main/models.py"
    click TOOLS "https://github.com/nm512/dreamerv3-torch/blob/main/tools.py"
    click DREAM "https://github.com/nm512/dreamerv3-torch/blob/main/dreamer.py"
    click IMGS "https://github.com/nm512/dreamerv3-torch/tree/main/imgs/"

    %% Styles
    classDef compute fill:#ADD8E6,stroke:#333,stroke-width:1px;
    classDef data fill:#90EE90,stroke:#333,stroke-width:1px;
    classDef external fill:#FFA500,stroke:#333,stroke-width:1px;
    classDef config fill:#DDA0DD,stroke:#333,stroke-width:1px;
```
