How denis-samatov/rl_course works
Project Type and Purpose
• This repository is an open-source educational toolkit—a practical course in Reinforcement Learning (RL) with code examples, Jupyter notebooks, and theoretical notes.
• Its goal is to teach key RL algorithms (MDP, Monte Carlo, TD, Q-learning, DQN, PPO, DPO, RLHF) through code demonstrations and mini-projects.
File Structure Analysis
• Top-level directories:
• code/ – per-seminar modules (08_mc_vs_td, 09_q_learning_bellman, …, 17_final_project), each containing scripts (*.py), notebooks (*.ipynb), and a seminar README.
• notes/ – theoretical materials: Markdown notes, PDFs, presentations, images.
• Root files: README.md, .gitignore, requirements.txt.
• No central src/ or microservice split; instead, each seminar fully self-contained.
• Configuration: requirements.txt lists Python dependencies; no Docker or CI scripts.
Key Insights from README
• Tech stack: Python 3.10, Gymnasium (RL environments), PyTorch, NumPy, Matplotlib, stable-baselines3, trl, wandb.
• Execution: Virtual environment (venv), CLI scripts, Jupyter Lab.
• Seminar pattern: theory → code → discussion.
a. Main Components
• Student/User Interface: Jupyter Lab / CLI (runs notebooks and scripts).
• Code Modules: Seminar folders under code/, each with algorithm implementations and demos.
• Theoretical Assets: notes/ (Markdown, PDFs, presentations).
• Python Environment: venv with dependencies from requirements.txt.
• External Services:
• Gymnasium environments (e.g., CartPole, LunarLander)
• wandb for experiment logging
• GitHub (code hosting)
b. Relationships and Data Flow
c. Architectural Patterns and Principles
• Modular per-seminar structure: each topic isolated, promoting reuse of pattern.
• Layered separation:
1. Presentation (notebooks)
2. Algorithm implementation (scripts)
3. Environment and utilities (Gym + helpers)
• Dependency isolation via virtual environment.
d. Key Technologies/Frameworks
• Python 3.10 interpreter
• Gymnasium (RL environments)
• PyTorch (neural nets in DQN, PPO, DPO)
• stable-baselines3, trl (for PPO, DPO implementations)
• wandb (experiment tracking)
• Jupyter Lab (interactive execution)
Final Recommendations
• Be exhaustive: list every seminar folder, show its internal scripts and notebooks.
• Execution environment: the venv sits between user tools and code modules.