# denis-samatov/rl_course architecture

> Architecture diagram of the GitHub repository denis-samatov/rl_course, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/denis-samatov/rl_course
- Repository: https://github.com/denis-samatov/rl_course

## Overview

##### Project Type and Purpose
- This repository is an open-source educational toolkit—a practical course in Reinforcement Learning (RL) with code examples, Jupyter notebooks, and theoretical notes.
- Its goal is to teach key RL algorithms (MDP, Monte Carlo, TD, Q-learning, DQN, PPO, DPO, RLHF) through code demonstrations and mini-projects.

##### File Structure Analysis
- Top-level directories:
- `code/` – per-seminar modules (08_mc_vs_td, 09_q_learning_bellman, …, 17_final_project), each containing scripts (`*.py`), notebooks (`*.ipynb`), and a seminar README.
- `notes/` – theoretical materials: Markdown notes, PDFs, presentations, images.
- Root files: `README.md`, `.gitignore`, `requirements.txt`.
- No central `src/` or microservice split; instead, each seminar fully self-contained.
- Configuration: `requirements.txt` lists Python dependencies; no Docker or CI scripts.

##### Key Insights from README
- Tech stack: Python 3.10, Gymnasium (RL environments), PyTorch, NumPy, Matplotlib, stable-baselines3, trl, wandb.
- Execution: Virtual environment (`venv`), CLI scripts, Jupyter Lab.
- Seminar pattern: theory → code → discussion.

a. Main Components
- Student/User Interface: Jupyter Lab / CLI (runs notebooks and scripts).
- Code Modules: Seminar folders under `code/`, each with algorithm implementations and demos.
- Theoretical Assets: `notes/` (Markdown, PDFs, presentations).
- Python Environment: `venv` with dependencies from `requirements.txt`.
- External Services:
- Gymnasium environments (e.g., CartPole, LunarLander)
- wandb for experiment logging
- GitHub (code hosting)

b. Relationships and Data Flow

c. Architectural Patterns and Principles
- Modular per-seminar structure: each topic isolated, promoting reuse of pattern.
- Layered separation:
1. Presentation (notebooks)
2. Algorithm implementation (scripts)
3. Environment and utilities (Gym + helpers)
- Dependency isolation via virtual environment.

d. Key Technologies/Frameworks
- Python 3.10 interpreter
- Gymnasium (RL environments)
- PyTorch (neural nets in DQN, PPO, DPO)
- stable-baselines3, trl (for PPO, DPO implementations)
- wandb (experiment tracking)
- Jupyter Lab (interactive execution)

##### Final Recommendations
- Be exhaustive: list every seminar folder, show its internal scripts and notebooks.
- Execution environment: the `venv` sits between user tools and code modules.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interface
    Student["Student/User"]:::ui
    Jupyter["Jupyter Lab"]:::ui
    CLI["CLI"]:::ui

    Student -->|interacts with| Jupyter
    Student -->|interacts with| CLI

    %% Virtual Environment and Config
    Requirements["requirements.txt"]:::config
    Gitignore[".gitignore"]:::config
    Venv["venv (Python 3.10)"]:::config
    README_Root["README.md"]:::docs

    Requirements -->|installs deps into| Venv
    Venv -->|provides runtime for| Jupyter
    Venv -->|provides runtime for| CLI

    Student -->|references| README_Root

    %% Code Modules
    subgraph "code/" 
        direction TB
        Seminar08["Seminar 08: MC vs TD"]:::component
        Seminar09["Seminar 09: Q-Learning & Bellman"]:::component
        Seminar10["Seminar 10: Deep Q-Network"]:::component
        Seminar11["Seminar 11: Policy Gradient (REINFORCE)"]:::component
        Seminar12["Seminar 12: Actor-Critic (A2C)"]:::component
        Seminar13["Seminar 13: Dynamic Programming"]:::component
        Seminar14["Seminar 14: PPO & TRPO"]:::component
        Seminar15["Seminar 15: RLHF Basics"]:::component
        Seminar16["Seminar 16: Direct Preference Optimization"]:::component
        Seminar17["Seminar 17: Final Project"]:::component
    end

    Jupyter -->|runs notebooks/scripts| Seminar08
    Jupyter -->|runs notebooks/scripts| Seminar09
    Jupyter -->|runs notebooks/scripts| Seminar10
    Jupyter -->|runs notebooks/scripts| Seminar11
    Jupyter -->|runs notebooks/scripts| Seminar12
    Jupyter -->|runs notebooks/scripts| Seminar13
    Jupyter -->|runs notebooks/scripts| Seminar14
    Jupyter -->|runs notebooks/scripts| Seminar15
    Jupyter -->|runs notebooks/scripts| Seminar16
    Jupyter -->|runs notebooks/scripts| Seminar17

    CLI -->|executes Python scripts| Seminar08
    CLI -->|executes Python scripts| Seminar09
    CLI -->|executes Python scripts| Seminar10
    CLI -->|executes Python scripts| Seminar11
    CLI -->|executes Python scripts| Seminar12
    CLI -->|executes Python scripts| Seminar13
    CLI -->|executes Python scripts| Seminar14
    CLI -->|executes Python scripts| Seminar15
    CLI -->|executes Python scripts| Seminar16
    CLI -->|executes Python scripts| Seminar17

    click Seminar08 "https://github.com/denis-samatov/rl_course/tree/main/code/08_mc_vs_td"
    click Seminar09 "https://github.com/denis-samatov/rl_course/tree/main/code/09_q_learning_bellman"
    click Seminar10 "https://github.com/denis-samatov/rl_course/tree/main/code/10_deep_q_network"
    click Seminar11 "https://github.com/denis-samatov/rl_course/tree/main/code/11_policy_gradient"
    click Seminar12 "https://github.com/denis-samatov/rl_course/tree/main/code/12_actor_critic"
    click Seminar13 "https://github.com/denis-samatov/rl_course/tree/main/code/13_dynamic_programming"
    click Seminar14 "https://github.com/denis-samatov/rl_course/tree/main/code/14_ppo_trpo"
    click Seminar15 "https://github.com/denis-samatov/rl_course/tree/main/code/15_rlhf_basics"
    click Seminar16 "https://github.com/denis-samatov/rl_course/tree/main/code/16_dpo"
    click Seminar17 "https://github.com/denis-samatov/rl_course/tree/main/code/17_final_project"

    %% Theoretical Assets
    subgraph "notes/" 
        direction TB
        NotesMD["Markdown Notes"]:::docs
        NotesPDF["PDF Notes"]:::docs
        NotesPres["Presentations"]:::docs
        NotesReadme["notes/README.md"]:::docs
    end

    Student -->|reviews theory| NotesMD
    Student -->|reviews theory| NotesPDF
    Student -->|reviews theory| NotesPres
    Student -->|reviews theory| NotesReadme

    click NotesMD "https://github.com/denis-samatov/rl_course/tree/main/notes/md"
    click NotesPDF "https://github.com/denis-samatov/rl_course/tree/main/notes/pdf"
    click NotesPres "https://github.com/denis-samatov/rl_course/tree/main/notes/presentation"
    click NotesReadme "https://github.com/denis-samatov/rl_course/blob/main/notes/README.md"

    %% External Services
    Gym["Gymnasium Environments"]:::external
    Wandb["wandb API"]:::external
    GitHub["GitHub (code hosting)"]:::external

    Seminar08 -->|environment interaction| Gym
    Seminar09 -->|environment interaction| Gym
    Seminar10 -->|environment interaction| Gym
    Seminar11 -->|environment interaction| Gym
    Seminar12 -->|environment interaction| Gym
    Seminar13 -->|environment interaction| Gym
    Seminar14 -->|environment interaction| Gym
    Seminar15 -->|environment interaction| Gym
    Seminar16 -->|environment interaction| Gym
    Seminar17 -->|environment interaction| Gym

    Seminar08 -->|logs metrics| Wandb
    Seminar09 -->|logs metrics| Wandb
    Seminar10 -->|logs metrics| Wandb
    Seminar11 -->|logs metrics| Wandb
    Seminar12 -->|logs metrics| Wandb
    Seminar13 -->|logs metrics| Wandb
    Seminar14 -->|logs metrics| Wandb
    Seminar15 -->|logs metrics| Wandb
    Seminar16 -->|logs metrics| Wandb
    Seminar17 -->|logs metrics| Wandb

    Seminar08 -->|saves figures/models| FileSystem["Local Filesystem"]:::component
    Seminar09 -->|saves figures/models| FileSystem
    Seminar10 -->|saves figures/models| FileSystem
    Seminar11 -->|saves figures/models| FileSystem
    Seminar12 -->|saves figures/models| FileSystem
    Seminar13 -->|saves figures/models| FileSystem
    Seminar14 -->|saves figures/models| FileSystem
    Seminar15 -->|saves figures/models| FileSystem
    Seminar16 -->|saves figures/models| FileSystem
    Seminar17 -->|saves figures/models| FileSystem

    Student -->|clones repo| GitHub

    %% Styling
    classDef component fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef ui fill:#FFA500,stroke:#333,stroke-width:1px
    classDef external fill:#90EE90,stroke:#333,stroke-width:1px
    classDef config fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef docs fill:#FFE4B5,stroke:#333,stroke-width:1px
```
