# yewr/efficientzero architecture

> Architecture diagram of the GitHub repository yewr/efficientzero, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yewr/efficientzero
- Repository: https://github.com/yewr/efficientzero

## Overview

##### Project Type and Purpose
- This is a full-stack reinforcement-learning training pipeline (“EfficientZero”) for Atari games.
- It combines Python orchestration (main.py, core/…), a C++/Cython MCTS engine (core/ctree), PyTorch models, and Ray-based distributed workers.

##### File Structure
Top-level directories:
- config/atari – domain configs & wrappers (defines BaseConfig, model hyperparameters, env wrappers)
- core/ctree – C++ + Cython extension for Monte Carlo Tree Search
- core/*.py – Python modules: dataset, game logic, MCTS, model definition, replay buffer, self-play & reanalyze workers, training, logging, storage
- main.py – CLI entrypoint; parses args, registers environment case, launches Ray actors
Shell scripts: train.sh, test.sh

##### README for Additional Insights
- Confirms distributed architecture on Ray: CPU actors for self-play, GPU actors for training, optional reanalysis workers.
- External dependencies: Python ≥3.6, PyTorch ≥1.8, GCC ≥7.5, Ray.

##### Main Components & Their Interactions
a. Configuration Loader
– Reads config/atari/*.py, instantiates environment wrappers and model architecture.
b. Ray Cluster
– Head node (main.py)
– CPU Actors (core.selfplay_worker.py) run multiple environment instances, call MCTS via core/ctree to generate trajectories.
– Replay Buffer / Storage (core.replay_buffer.py & core.storage.py) collects self-play data.
– GPU Actors (core.train.py) sample minibatches from replay buffer, perform gradient updates on the PyTorch model (core.model.py).
– Reanalyze Workers (core.reanalyze_worker.py) periodically pull past trajectories, re-run MCTS with the latest model to refine targets.
c. C++/Cython MCTS Engine
– Exposed to Python via cytree.pyx, built by core/ctree/make.sh; used by both self-play and reanalysis.
d. Logging & Checkpointing
– core.log.py for metrics
– model saves via core/storage.py

##### Architectural Patterns & Design Principles
- Actor-based concurrency (Ray remote functions) – isolates self-play, training, reanalysis into separate processes.
- Data-driven configuration – new domains plug-in under config/, no code changes to core pipeline.
- Modular MCTS engine – compiled extension for performance.
- Separation of concerns: environment simulators (Gym wrappers), search (ctree), learning (PyTorch), data storage.

##### Final Rendering Checklist
– Note any loops (self-play → train → reanalyze → self-play) to capture the continual learning cycle

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Orchestration & Config
    subgraph "Orchestration & Config"
        Main["Main Orchestrator\n(main.py)"]:::orch
        subgraph "Configuration Loader"
            CoreConfig["core/config.py"]:::orch
            EnvWrapper["config/atari/env_wrapper.py"]:::orch
            ModelConf["config/atari/model.py"]:::orch
        end
        ShellTrain["train.sh"]:::orch
        ShellTest["test.sh"]:::orch
    end

    %% Ray Cluster
    subgraph "Ray Cluster"
        SelfPlay["CPU Actor\nSelf-Play Worker"]:::compute
        Trainer["GPU Actor\nTrainer"]:::compute
        Reanalyze["Reanalyze Worker"]:::compute
    end

    %% Data Stores and Services
    Replay["Replay Buffer"]:::data
    Storage["Storage & Checkpointing"]:::data
    Log["Logging / Metrics"]:::data
    Model["PyTorch Model"]:::data
    MCTSIf["MCTS Interface\n(core/mcts.py)"]:::compile
    Engine["C++/Cython MCTS Engine"]:::compile
    Env["Atari Environment\n(Gym Wrapper)"]:::external

    %% Support Modules
    GameLogic["Game Logic\n(core/game.py)"]:::orch
    Dataset["Dataset Utils\n(core/dataset.py)"]:::orch
    Utils["Utilities\n(core/utils.py)"]:::orch
    TestEntry["Test Entrypoint\n(core/test.py)"]:::orch

    %% Connections
    Main --> CoreConfig
    Main --> EnvWrapper
    Main --> ModelConf
    Main --> ShellTrain
    Main --> ShellTest
    Main --> SelfPlay
    Main --> Trainer
    Main --> Reanalyze

    SelfPlay -->|"calls"| MCTSIf
    MCTSIf -->|"executes"| Engine
    Engine -.->|"simulates"| Env
    Env -->|"trajectories"| Replay
    SelfPlay -->|"uses"| GameLogic
    SelfPlay -->|"uses"| Dataset
    SelfPlay --> Log

    Trainer -->|"pulls minibatches"| Replay
    Trainer -->|"updates"| Model
    Model -->|"checkpoint"| Storage
    Trainer --> Log

    Reanalyze -->|"reads old trajectories"| Replay
    Reanalyze -->|"calls"| MCTSIf
    Reanalyze -->|"writes refined targets"| Replay
    Reanalyze --> Log

    Storage --> Log

    %% Click Events
    click Main "https://github.com/yewr/efficientzero/blob/main/main.py"
    click CoreConfig "https://github.com/yewr/efficientzero/blob/main/core/config.py"
    click EnvWrapper "https://github.com/yewr/efficientzero/blob/main/config/atari/env_wrapper.py"
    click ModelConf "https://github.com/yewr/efficientzero/blob/main/config/atari/model.py"
    click ShellTrain "https://github.com/yewr/efficientzero/blob/main/train.sh"
    click ShellTest "https://github.com/yewr/efficientzero/blob/main/test.sh"
    click SelfPlay "https://github.com/yewr/efficientzero/blob/main/core/selfplay_worker.py"
    click Trainer "https://github.com/yewr/efficientzero/blob/main/core/train.py"
    click Reanalyze "https://github.com/yewr/efficientzero/blob/main/core/reanalyze_worker.py"
    click Replay "https://github.com/yewr/efficientzero/blob/main/core/replay_buffer.py"
    click Storage "https://github.com/yewr/efficientzero/blob/main/core/storage.py"
    click Log "https://github.com/yewr/efficientzero/blob/main/core/log.py"
    click Model "https://github.com/yewr/efficientzero/blob/main/core/model.py"
    click MCTSIf "https://github.com/yewr/efficientzero/blob/main/core/mcts.py"
    click Engine "https://github.com/yewr/efficientzero/tree/main/core/ctree"
    click GameLogic "https://github.com/yewr/efficientzero/blob/main/core/game.py"
    click Dataset "https://github.com/yewr/efficientzero/blob/main/core/dataset.py"
    click Utils "https://github.com/yewr/efficientzero/blob/main/core/utils.py"
    click TestEntry "https://github.com/yewr/efficientzero/blob/main/core/test.py"

    %% Styles
    classDef orch fill:#D6E9FE,stroke:#000
    classDef compute fill:#D6F5D6,stroke:#000
    classDef data fill:#FFE5B4,stroke:#000
    classDef compile fill:#D3D3D3,stroke:#000
    classDef external fill:#F0F0F0,stroke:#000
```
