# werner-duvaud/muzero-general architecture

> Architecture diagram of the GitHub repository werner-duvaud/muzero-general, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/werner-duvaud/muzero-general
- Repository: https://github.com/werner-duvaud/muzero-general

## Overview

##### Project type and purpose
- This is a research-style, full-stack reinforcement-learning framework implementing the MuZero algorithm in Python.
- Purpose: Train and evaluate MuZero on arbitrary games/environments by plugging in a game class (under `games/`), then running self-play, training, checkpointing, and TensorBoard logging.

##### File structure
- Top-level scripts:
- `muzero.py` – entrypoint & orchestration
- `self_play.py` – worker logic for generating game trajectories
- `trainer.py` – worker logic for consuming trajectories and updating the model
- `shared_storage.py` – central store for model weights, training statistics, hyperparameters
- `replay_buffer.py` – queue of past trajectories for training
- `models.py` – neural-network definitions (representation, dynamics, prediction heads)
- `diagnose_model.py` – auxiliary tool for model introspection
- `games/` folder: one file per environment/game, each defining a `MuZeroConfig` subclass and environment interface
- `results/`: TensorBoard logs and model checkpoints, organized per game
- Configuration files (.github workflows, requirements)

##### Key insights from the README
- Uses PyTorch for networks; Ray for asynchronous multi-thread or cluster execution; multi-GPU support.
- Major components: Self-play workers, training workers, shared storage, replay buffer, model.
- Data flow: workers ↔ shared storage ↔ replay buffer → trainer → checkpoints & logs.

## Diagram (Mermaid)

```mermaid
graph TB
    %% External Services
    subgraph "External Services"
        Ray["Ray Cluster"]:::external
        GPU["GPU Pool"]:::external
        TB["TensorBoard"]:::external
        FS["Filesystem (Checkpoints)"]:::external
    end

    %% State Layer
    subgraph "State"
        SS[(Shared Storage)]:::datastore
        RB[(Replay Buffer)]:::datastore
        RS[(Results Store)]:::datastore
        DOCS["Documentation"]:::datastore
        CI["CI Workflows"]:::datastore
        DEPS["Dependencies"]:::datastore
    end

    %% Compute Layer
    subgraph "Compute"
        Entrypoint["Entrypoint\n(muzero.py)"]:::process
        subgraph "Self-Play Actors"
            SP[/"n× Self-Play Actor"/]:::process
        end
        subgraph "Trainer Actors"
            TR[/"m× Trainer Actor"/]:::process
        end
        DIAG["Diagnostics Tool\n(diagnose_model.py)"]:::process
    end

    %% Model & Games
    subgraph "Core Modules"
        MOD["Model Definitions\n(models.py)"]:::process
        AG["Game Interface\n(abstract_game.py)"]:::process
        subgraph "Game Configs"
            G1["atari.py"]:::process
            G2["breakout.py"]:::process
            G3["cartpole.py"]:::process
            G4["connect4.py"]:::process
            G5["gomoku.py"]:::process
            G6["gridworld.py"]:::process
            G7["lunarlander.py"]:::process
            G8["simple_grid.py"]:::process
            G9["spiel.py"]:::process
            G10["tictactoe.py"]:::process
            G11["twentyone.py"]:::process
        end
    end

    %% Orchestration and Flow
    Entrypoint -->|reads config| AG
    Entrypoint -->|launch actors via Ray| Ray
    Ray --> SP
    Ray --> TR
    SP -->|request weights| SS
    SS -->|send latest model| SP
    SP -->|write trajectories| RB
    TR -->|read batches| RB
    TR -->|train via| MOD
    TR -->|update weights| SS
    TR -->|log metrics| TB
    TR -->|write checkpoints| FS
    FS --> RS
    DIAG --> MOD

    %% Click Events
    click Entrypoint "https://github.com/werner-duvaud/muzero-general/blob/master/muzero.py"
    click AG "https://github.com/werner-duvaud/muzero-general/blob/master/games/abstract_game.py"
    click G1 "https://github.com/werner-duvaud/muzero-general/blob/master/games/atari.py"
    click G2 "https://github.com/werner-duvaud/muzero-general/blob/master/games/breakout.py"
    click G3 "https://github.com/werner-duvaud/muzero-general/blob/master/games/cartpole.py"
    click G4 "https://github.com/werner-duvaud/muzero-general/blob/master/games/connect4.py"
    click G5 "https://github.com/werner-duvaud/muzero-general/blob/master/games/gomoku.py"
    click G6 "https://github.com/werner-duvaud/muzero-general/blob/master/games/gridworld.py"
    click G7 "https://github.com/werner-duvaud/muzero-general/blob/master/games/lunarlander.py"
    click G8 "https://github.com/werner-duvaud/muzero-general/blob/master/games/simple_grid.py"
    click G9 "https://github.com/werner-duvaud/muzero-general/blob/master/games/spiel.py"
    click G10 "https://github.com/werner-duvaud/muzero-general/blob/master/games/tictactoe.py"
    click G11 "https://github.com/werner-duvaud/muzero-general/blob/master/games/twentyone.py"
    click SP "https://github.com/werner-duvaud/muzero-general/blob/master/self_play.py"
    click TR "https://github.com/werner-duvaud/muzero-general/blob/master/trainer.py"
    click SS "https://github.com/werner-duvaud/muzero-general/blob/master/shared_storage.py"
    click RB "https://github.com/werner-duvaud/muzero-general/blob/master/replay_buffer.py"
    click MOD "https://github.com/werner-duvaud/muzero-general/blob/master/models.py"
    click DIAG "https://github.com/werner-duvaud/muzero-general/blob/master/diagnose_model.py"
    click RS "https://github.com/werner-duvaud/muzero-general/blob/master/results/cartpole/model.checkpoint"
    click DOCS "https://github.com/werner-duvaud/muzero-general/blob/master/docs/README.md"
    click CI "https://github.com/werner-duvaud/muzero-general/blob/master/.github/workflows/ci-testing.yaml"
    click DEPS "https://github.com/werner-duvaud/muzero-general/blob/master/requirements.txt"

    %% Styles
    classDef process fill:#9f6,stroke:#333,stroke-width:1px
    classDef datastore fill:#6cf,stroke:#333,stroke-width:1px
    classDef external fill:#f96,stroke:#333,stroke-width:1px
```
