How applied-machine-learning-lab/rmtl works
Project type and purpose
• This is a research‐oriented machine‐learning codebase for multi‐task recommendation. It implements both supervised learning (SL) baselines (in slmodels) and reinforcement learning (RL) agents (agents), all sharing low‐level building blocks (layers). It trains and evaluates models offline using session‐based RetailRocket data via a simulated environment (env.py).
File structure
• Top‐level scripts:
• SLmain.py – entry point for supervised‐learning baselines
• RLmain.py – entry point for RL training
• Core modules:
• layers/ – common network building blocks (embeddings, MLPs, actor & critic definitions)
• slmodels/ – supervised multi‐task models (ESMM, MMOE, PLE, shared‐bottom, etc.)
• agents/ – RL agents packaging policy (actor) & critic networks plus experience replay
• train/ – argument parsing, run orchestration, utility functions
• env.py – offline simulation environment turning dataset logs into MDP samples
• Config & docs: README.md, Framework.pdf, doc.md, pretrain.zip
README for additional insights
• The README outlines the two modes (SL vs. RL), points to dataset format, and shows command‐line usage. It also explains how the ESMM actor network can embed any SL model from slmodels as its policy.
a. Main components to represent:
1. Data Layer:
– Raw RetailRocket logs (CSV or preprocessed MDP format)
– Preprocessing & dataset loader (embedded in env.py)
2. Simulation Environment (env.py):
– Episodic sampler converting sessions into state, action, reward tuples
3. Model Layer:
– SL Baselines (slmodels/…)
– RL Agents (agents/…):
• Actor network (ESMM + optional SL model)
• Critic network
• Replay buffer (ReplayBuffer)
4. Training Orchestration (train/run.py + Arguments.py + utils.py):
– Hyperparameter parsing
– Training loops (SL vs. RL)
– Evaluation & metric logging
5. Entry‐point scripts:
– SLmain.py
– RLmain.py
6. Outputs & Monitoring:
– AUC, log‐loss metrics
– Model checkpoints (optional)
– Console or file‐based logs
b. Relationships & dataflow:
– Environment ↔ Agents (state → agent → action, then reward back to agent)
– Agents call layers/ and slmodels/ to build networks
– Training orchestrator invokes either SLmain.py or RLmain.py, passing through train/run.py
– Metrics from training flow back to orchestrator, then to logs or visualization
c. Architectural patterns & design principles:
– Modular separation: data (env), model definitions (layers/slmodels/agents), training logic (train/)
– Pluggable policy architecture: ESMM actor can swap in any slmodels implementation
– Clear distinction between SL and RL modes—shared building blocks for consistency
d. Technology calls‐outs:
– Deep‐learning framework (e.g., PyTorch or TensorFlow—implied by layers/*.py)
– CLI driven with train/Arguments.py (argparse)
– Offline simulation (no live A/B service)