How yewr/efficientzero works
Project Type and Purpose
• This is a full-stack reinforcement-learning training pipeline (“EfficientZero”) for Atari games.
• It combines Python orchestration (main.py, core/…), a C++/Cython MCTS engine (core/ctree), PyTorch models, and Ray-based distributed workers.
File Structure
Top-level directories:
• config/atari – domain configs & wrappers (defines BaseConfig, model hyperparameters, env wrappers)
• core/ctree – C++ + Cython extension for Monte Carlo Tree Search
• core/*.py – Python modules: dataset, game logic, MCTS, model definition, replay buffer, self-play & reanalyze workers, training, logging, storage
• main.py – CLI entrypoint; parses args, registers environment case, launches Ray actors
Shell scripts: train.sh, test.sh
README for Additional Insights
• Confirms distributed architecture on Ray: CPU actors for self-play, GPU actors for training, optional reanalysis workers.
• External dependencies: Python ≥3.6, PyTorch ≥1.8, GCC ≥7.5, Ray.
Main Components & Their Interactions
a. Configuration Loader
– Reads config/atari/*.py, instantiates environment wrappers and model architecture.
b. Ray Cluster
– Head node (main.py)
– CPU Actors (core.selfplay_worker.py) run multiple environment instances, call MCTS via core/ctree to generate trajectories.
– Replay Buffer / Storage (core.replay_buffer.py & core.storage.py) collects self-play data.
– GPU Actors (core.train.py) sample minibatches from replay buffer, perform gradient updates on the PyTorch model (core.model.py).
– Reanalyze Workers (core.reanalyze_worker.py) periodically pull past trajectories, re-run MCTS with the latest model to refine targets.
c. C++/Cython MCTS Engine
– Exposed to Python via cytree.pyx, built by core/ctree/make.sh; used by both self-play and reanalysis.
d. Logging & Checkpointing
– core.log.py for metrics
– model saves via core/storage.py
Architectural Patterns & Design Principles
• Actor-based concurrency (Ray remote functions) – isolates self-play, training, reanalysis into separate processes.
• Data-driven configuration – new domains plug-in under config/, no code changes to core pipeline.
• Modular MCTS engine – compiled extension for performance.
• Separation of concerns: environment simulators (Gym wrappers), search (ctree), learning (PyTorch), data storage.
Final Rendering Checklist
– Note any loops (self-play → train → reanalyze → self-play) to capture the continual learning cycle