How kakaoenterprise/jorldy works
Project type and purpose
• JORLDY is an open-source, framework-style RL toolkit (not a web app). It exposes a CLI entrypoint (main.py) and lets users pick/train/test 20+ RL algorithms on various environments (Gym, Atari, MuJoCo, Unity ML-Agents). It also supports distributed training via Ray.
High-level components
a. Entrypoint / CLI
– main.py, run_mode.py, process.py parse CLI flags (--config, --async/--sync), bootstrap the system.
b. Configuration subsystem
– jorldy/config: per-algorithm & per-environment Python modules.
– config_manager (in jorldy/manager) dynamically loads hyperparameters & env settings.
c. Manager layer
– distributed_manager (wraps Ray actors & remote workers).
– eval_manager (periodic evaluation).
– log_manager (tensorboard, file logs).
– metric_manager (aggregates performance stats).
d. Core RL pipeline
– Agent (jorldy/core/agent): policy definitions, selects actions given states.
– Environment (jorldy/core/env): wrappers over Gym, MuJoCo, Atari, Unity ML-Agents binaries.
– Buffer (jorldy/core/buffer): replay buffers, rollout buffers, PER buffers.
– Network (jorldy/core/network): neural net architectures, heads, policies, Q-networks.
– Optimizer (jorldy/core/optimizer): optimizers and learning-rate schedules.
e. External systems / libs
– Ray for distribution.
– Gym / gym[atari] / gym-super-mario-bros.
– MuJoCo-py, Unity ML-Agents executables.
– Docker (in docker/Dockerfile).
Data flow & interactions
1) CLI → Config manager loads <algorithm>.<env> module → Instantiates Env, Agent, Buffer, Network, Optimizer.
2) If --async: Distributed manager spawns remote rollout & learner workers; otherwise single‐process loop.
3) Rollout loop:
– Environment emits state → Agent.query(policy network) → Action → Environment step → next state & reward.
– Transition pushed into Buffer.
4) Learning loop:
– Optimizer samples from Buffer → forward pass through Network → compute gradients → update Network parameters.
– Agent’s policy network is updated (in-place or via weight sync messages in distributed mode).
5) Managers:
– Log manager records loss, rewards.
– Metric manager aggregates and visualizes.
– Eval manager periodically runs deterministic episodes.
6) Distributed manager (if used) orchestrates Ray remote calls, parameter broadcasting, worker health.
Architectural patterns & principles
• Plugin architecture: algorithms, envs, networks discovered via Python dicts (_agent_dict.txt, _env_dict.txt, _network_dict.txt, _optimizer_dict.txt).
• Separation of concerns: managers handle orchestration, core modules handle domain logic (env/agent/network/optimizer).
• Config-driven instantiation: all hyperparameters reside in jorldy/config/*.
• Support for both synchronous and asynchronous (actor-learner) training.