# kakaoenterprise/jorldy architecture

> Architecture diagram of the GitHub repository kakaoenterprise/jorldy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kakaoenterprise/jorldy
- Repository: https://github.com/kakaoenterprise/jorldy

## Overview

##### Project type and purpose
- JORLDY is an open-source, framework-style RL toolkit (not a web app). It exposes a CLI entrypoint (`main.py`) and lets users pick/train/test 20+ RL algorithms on various environments (Gym, Atari, MuJoCo, Unity ML-Agents). It also supports distributed training via Ray.

##### High-level components
a. Entrypoint / CLI
– `main.py`, `run_mode.py`, `process.py` parse CLI flags (`--config`, `--async`/`--sync`), bootstrap the system.
b. Configuration subsystem
– `jorldy/config`: per-algorithm & per-environment Python modules.
– `config_manager` (in `jorldy/manager`) dynamically loads hyperparameters & env settings.
c. Manager layer
– `distributed_manager` (wraps Ray actors & remote workers).
– `eval_manager` (periodic evaluation).
– `log_manager` (tensorboard, file logs).
– `metric_manager` (aggregates performance stats).
d. Core RL pipeline
– Agent (`jorldy/core/agent`): policy definitions, selects actions given states.
– Environment (`jorldy/core/env`): wrappers over Gym, MuJoCo, Atari, Unity ML-Agents binaries.
– Buffer (`jorldy/core/buffer`): replay buffers, rollout buffers, PER buffers.
– Network (`jorldy/core/network`): neural net architectures, heads, policies, Q-networks.
– Optimizer (`jorldy/core/optimizer`): optimizers and learning-rate schedules.
e. External systems / libs
– Ray for distribution.
– Gym / gym[atari] / gym-super-mario-bros.
– MuJoCo-py, Unity ML-Agents executables.
– Docker (in `docker/Dockerfile`).

##### Data flow & interactions
1) CLI → Config manager loads `<algorithm>.<env>` module → Instantiates Env, Agent, Buffer, Network, Optimizer.
2) If `--async`: Distributed manager spawns remote rollout & learner workers; otherwise single‐process loop.
3) Rollout loop:
– Environment emits state → Agent.query(policy network) → Action → Environment step → next state & reward.
– Transition pushed into Buffer.
4) Learning loop:
– Optimizer samples from Buffer → forward pass through Network → compute gradients → update Network parameters.
– Agent’s policy network is updated (in-place or via weight sync messages in distributed mode).
5) Managers:
– Log manager records loss, rewards.
– Metric manager aggregates and visualizes.
– Eval manager periodically runs deterministic episodes.
6) Distributed manager (if used) orchestrates Ray remote calls, parameter broadcasting, worker health.

##### Architectural patterns & principles
- Plugin architecture: algorithms, envs, networks discovered via Python dicts (`_agent_dict.txt`, `_env_dict.txt`, `_network_dict.txt`, `_optimizer_dict.txt`).
- Separation of concerns: managers handle orchestration, core modules handle domain logic (env/agent/network/optimizer).
- Config-driven instantiation: all hyperparameters reside in `jorldy/config/*`.
- Support for both synchronous and asynchronous (actor-learner) training.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI and Configuration Layer
    subgraph "CLI / Config Layer"
        direction TB
        main["main.py"]:::cli
        run_mode["run_mode.py"]:::cli
        process["process.py"]:::cli
        config_manager["Config Manager"]:::cli
        config_dir["jorldy/config/"]:::cli
    end

    %% Manager Layer
    subgraph "Manager Layer"
        direction TB
        distributed_manager["Distributed Manager (Ray)"]:::manager
        eval_manager["Eval Manager"]:::manager
        log_manager["Log Manager"]:::manager
        metric_manager["Metric Manager"]:::manager
    end

    %% Core RL Pipeline
    subgraph "Core RL Pipeline"
        direction TB
        subgraph "Plugin Registries"
            direction TB
            agent_registry["Agent Registry"]:::core
            env_registry["Environment Registry"]:::core
            network_registry["Network Registry"]:::core
            optimizer_registry["Optimizer Registry"]:::core
        end
        Agent["Agent"]:::core
        Env["Environment"]:::core
        Buffer[(Replay/ Rollout / PER Buffer)]:::core
        Network["Policy / Q-Network"]:::core
        Optimizer["Optimizer & Schedules"]:::core
    end

    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        Ray["Ray"]:::external
        Gym["Gym / Atari / Super Mario"]:::external
        MuJoCo["MuJoCo-py"]:::external
        UnityML["Unity ML-Agents"]:::external
        Docker["Dockerfile"]:::external
        Req["requirements.txt"]:::external
    end

    %% Connections
    main -->|parses flags| run_mode
    run_mode -->|bootstrap| process
    process -->|loads config| config_manager
    config_manager -->|reads configs| config_dir
    config_manager -->|instantiates| Env
    config_manager --> Agent
    config_manager --> Buffer
    config_manager --> Network
    config_manager --> Optimizer

    %% Distributed vs Single-process
    config_manager -->|--async| distributed_manager
    config_manager -->|--sync| Agent

    distributed_manager --> Ray
    Agent -->|rollout| Env
    Env -->|state| Agent
    Agent -->|action| Env
    Env -->|transition| Buffer
    Buffer -->|sample| Network
    Network -->|gradients| Optimizer
    Optimizer -->|update| Network
    Network -->|update policy| Agent

    eval_manager --> Agent
    eval_manager --> Env

    Agent --> log_manager
    Network --> log_manager
    Buffer --> metric_manager
    eval_manager --> metric_manager

    %% Plugin registries
    config_manager --> agent_registry
    config_manager --> env_registry
    config_manager --> network_registry
    config_manager --> optimizer_registry

    %% External library usage
    Env --> Gym
    Env --> MuJoCo
    Env --> UnityML

    %% Docker and requirements
    Docker -->|builds image| Ray
    Req -->|provides deps| process

    %% Click events
    click main "https://github.com/kakaoenterprise/jorldy/blob/master/jorldy/main.py"
    click run_mode "https://github.com/kakaoenterprise/jorldy/blob/master/jorldy/run_mode.py"
    click process "https://github.com/kakaoenterprise/jorldy/blob/master/jorldy/process.py"
    click config_manager "https://github.com/kakaoenterprise/jorldy/blob/master/jorldy/manager/config_manager.py"
    click config_dir "https://github.com/kakaoenterprise/jorldy/tree/master/jorldy/config/"
    click distributed_manager "https://github.com/kakaoenterprise/jorldy/blob/master/jorldy/manager/distributed_manager.py"
    click eval_manager "https://github.com/kakaoenterprise/jorldy/blob/master/jorldy/manager/eval_manager.py"
    click log_manager "https://github.com/kakaoenterprise/jorldy/blob/master/jorldy/manager/log_manager.py"
    click metric_manager "https://github.com/kakaoenterprise/jorldy/blob/master/jorldy/manager/metric_manager.py"
    click agent_registry "https://github.com/kakaoenterprise/jorldy/blob/master/jorldy/core/agent/_agent_dict.txt"
    click env_registry "https://github.com/kakaoenterprise/jorldy/blob/master/jorldy/core/env/_env_dict.txt"
    click network_registry "https://github.com/kakaoenterprise/jorldy/blob/master/jorldy/core/network/_network_dict.txt"
    click optimizer_registry "https://github.com/kakaoenterprise/jorldy/blob/master/jorldy/core/optimizer/_optimizer_dict.txt"
    click Docker "https://github.com/kakaoenterprise/jorldy/tree/master/docker/Dockerfile"

    %% Styles
    classDef cli fill:#f0f0f0,stroke:#333,stroke-width:1px;
    classDef manager fill:#bbdefb,stroke:#333,stroke-width:1px;
    classDef core fill:#c8e6c9,stroke:#333,stroke-width:1px;
    classDef external fill:#ffe0b2,stroke:#333,stroke-width:1px;
```
