# applied-machine-learning-lab/rmtl architecture

> Architecture diagram of the GitHub repository applied-machine-learning-lab/rmtl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/applied-machine-learning-lab/rmtl
- Repository: https://github.com/applied-machine-learning-lab/rmtl

## Overview

##### Project type and purpose
- This is a research‐oriented machine‐learning codebase for multi‐task recommendation. It implements both supervised learning (SL) baselines (in `slmodels`) and reinforcement learning (RL) agents (`agents`), all sharing low‐level building blocks (`layers`). It trains and evaluates models offline using session‐based RetailRocket data via a simulated environment (`env.py`).

##### File structure
- Top‐level scripts:
- `SLmain.py` – entry point for supervised‐learning baselines
- `RLmain.py` – entry point for RL training
- Core modules:
- `layers/` – common network building blocks (embeddings, MLPs, actor & critic definitions)
- `slmodels/` – supervised multi‐task models (ESMM, MMOE, PLE, shared‐bottom, etc.)
- `agents/` – RL agents packaging policy (actor) & critic networks plus experience replay
- `train/` – argument parsing, run orchestration, utility functions
- `env.py` – offline simulation environment turning dataset logs into MDP samples
- Config & docs: `README.md`, `Framework.pdf`, `doc.md`, `pretrain.zip`

##### README for additional insights
- The README outlines the two modes (SL vs. RL), points to dataset format, and shows command‐line usage. It also explains how the ESMM actor network can embed any SL model from `slmodels` as its policy.

a. Main components to represent:
1. Data Layer:
– Raw RetailRocket logs (CSV or preprocessed MDP format)
– Preprocessing & dataset loader (embedded in `env.py`)
2. Simulation Environment (`env.py`):
– Episodic sampler converting sessions into state, action, reward tuples
3. Model Layer:
– SL Baselines (`slmodels/…`)
– RL Agents (`agents/…`):
- Actor network (ESMM + optional SL model)
- Critic network
- Replay buffer (`ReplayBuffer`)
4. Training Orchestration (`train/run.py` + `Arguments.py` + `utils.py`):
– Hyperparameter parsing
– Training loops (SL vs. RL)
– Evaluation & metric logging
5. Entry‐point scripts:
– `SLmain.py`
– `RLmain.py`
6. Outputs & Monitoring:
– AUC, log‐loss metrics
– Model checkpoints (optional)
– Console or file‐based logs

b. Relationships & dataflow:
– Environment ↔ Agents (state → agent → action, then reward back to agent)
– Agents call `layers/` and `slmodels/` to build networks
– Training orchestrator invokes either `SLmain.py` or `RLmain.py`, passing through `train/run.py`
– Metrics from training flow back to orchestrator, then to logs or visualization

c. Architectural patterns & design principles:
– Modular separation: data (env), model definitions (layers/slmodels/agents), training logic (train/)
– Pluggable policy architecture: ESMM actor can swap in any `slmodels` implementation
– Clear distinction between SL and RL modes—shared building blocks for consistency

d. Technology calls‐outs:
– Deep‐learning framework (e.g., PyTorch or TensorFlow—implied by `layers/*.py`)
– CLI driven with `train/Arguments.py` (argparse)
– Offline simulation (no live A/B service)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Entry Points
    subgraph "Entry Points"
        SLmain["SLmain.py"]:::entry
        RLmain["RLmain.py"]:::entry
    end

    %% Training Orchestration
    subgraph "Training Orchestration"
        Args["Arguments.py"]:::orchestration
        Runner["run.py"]:::orchestration
        Utils["utils.py"]:::orchestration
    end

    %% Data & Environment
    subgraph "Data & Environment"
        Data["RetailRocket Logs (CSV/MDP)"]:::dataEnv
        Env["env.py"]:::dataEnv
    end

    %% Model Layer
    subgraph "Model Layer"
        Layers_core["layers/layers.py"]:::dataEnv
        Critic_layer["layers/critic.py"]:::dataEnv
        ESMM_layer["layers/esmm.py"]:::dataEnv
        subgraph "SL Models"
            aitm["aitm.py"]:::slModel
            esmm_sl["esmm.py"]:::slModel
            layers_sl["layers.py"]:::slModel
            metaheac["metaheac.py"]:::slModel
            mmoe["mmoe.py"]:::slModel
            omoe["omoe.py"]:::slModel
            ple["ple.py"]:::slModel
            sharedbottom["sharedbottom.py"]:::slModel
            singletask["singletask.py"]:::slModel
        end
    end

    %% RL Agents
    subgraph "RL Agents"
        DDPG1["DDPG_ESMM.py"]:::rlModel
        DDPG2["DDPG_ESMM_BC.py"]:::rlModel
        Replay["ReplayBuffer.py"]:::rlModel
    end

    %% Outputs & Monitoring
    subgraph "Outputs & Monitoring"
        Metrics["AUC, Log-loss, Rewards"]:::doc
        Checkpoints["Model Checkpoints & Logs"]:::doc
    end

    %% Connections
    SLmain -->|starts| Runner
    RLmain -->|starts| Runner
    Runner -->|parses args| Args
    Runner -->|loads utils| Utils
    Runner -->|loads data| Data
    Data -->|ingest| Env
    Runner -->|initializes env| Env

    %% SL flow
    Runner -->|trains SL| esmm_sl
    esmm_sl -->|uses| Layers_core
    esmm_sl -->|uses| layers_sl
    Runner -->|outputs metrics| Metrics

    %% RL flow
    Env -->|state| DDPG1
    Env -->|state| DDPG2
    DDPG1 -->|action| Env
    DDPG2 -->|action| Env
    Env -->|reward| DDPG1
    Env -->|reward| DDPG2
    DDPG1 -->|builds networks| ESMM_layer
    DDPG1 -->|builds critic| Critic_layer
    DDPG1 -->|stores experience| Replay
    DDPG2 -->|builds networks| ESMM_layer
    DDPG2 -->|builds critic| Critic_layer
    DDPG2 -->|stores experience| Replay
    Runner -->|orchestrates RL| DDPG1
    Runner -->|orchestrates RL| DDPG2
    Runner -->|outputs metrics| Checkpoints

    %% Click Events
    click SLmain "https://github.com/applied-machine-learning-lab/rmtl/blob/main/SLmain.py"
    click RLmain "https://github.com/applied-machine-learning-lab/rmtl/blob/main/RLmain.py"
    click Env "https://github.com/applied-machine-learning-lab/rmtl/blob/main/env.py"
    click Layers_core "https://github.com/applied-machine-learning-lab/rmtl/blob/main/layers/layers.py"
    click Critic_layer "https://github.com/applied-machine-learning-lab/rmtl/blob/main/layers/critic.py"
    click ESMM_layer "https://github.com/applied-machine-learning-lab/rmtl/blob/main/layers/esmm.py"
    click aitm "https://github.com/applied-machine-learning-lab/rmtl/blob/main/slmodels/aitm.py"
    click esmm_sl "https://github.com/applied-machine-learning-lab/rmtl/blob/main/slmodels/esmm.py"
    click layers_sl "https://github.com/applied-machine-learning-lab/rmtl/blob/main/slmodels/layers.py"
    click metaheac "https://github.com/applied-machine-learning-lab/rmtl/blob/main/slmodels/metaheac.py"
    click mmoe "https://github.com/applied-machine-learning-lab/rmtl/blob/main/slmodels/mmoe.py"
    click omoe "https://github.com/applied-machine-learning-lab/rmtl/blob/main/slmodels/omoe.py"
    click ple "https://github.com/applied-machine-learning-lab/rmtl/blob/main/slmodels/ple.py"
    click sharedbottom "https://github.com/applied-machine-learning-lab/rmtl/blob/main/slmodels/sharedbottom.py"
    click singletask "https://github.com/applied-machine-learning-lab/rmtl/blob/main/slmodels/singletask.py"
    click DDPG1 "https://github.com/applied-machine-learning-lab/rmtl/blob/main/agents/DDPG_ESMM.py"
    click DDPG2 "https://github.com/applied-machine-learning-lab/rmtl/blob/main/agents/DDPG_ESMM_BC.py"
    click Replay "https://github.com/applied-machine-learning-lab/rmtl/blob/main/agents/ReplayBuffer.py"
    click Args "https://github.com/applied-machine-learning-lab/rmtl/blob/main/train/Arguments.py"
    click Runner "https://github.com/applied-machine-learning-lab/rmtl/blob/main/train/run.py"
    click Utils "https://github.com/applied-machine-learning-lab/rmtl/blob/main/train/utils.py"

    %% Styles
    classDef dataEnv fill:#AEDFF7,stroke:#026494,stroke-width:1px;
    classDef slModel fill:#C6EFCF,stroke:#2E7D32,stroke-width:1px;
    classDef rlModel fill:#A3D5D3,stroke:#00695C,stroke-width:1px;
    classDef orchestration fill:#FFD59E,stroke:#EF6C00,stroke-width:1px;
    classDef entry fill:#E1BEE7,stroke:#6A1B9A,stroke-width:1px;
    classDef doc fill:#EEEEEE,stroke:#757575,stroke-width:1px;
```
