# lecar-lab/16831-f26-hw architecture

> Architecture diagram of the GitHub repository lecar-lab/16831-f26-hw, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lecar-lab/16831-f26-hw
- Repository: https://github.com/lecar-lab/16831-f26-hw

## Overview

This repository contains three iterations of robot-learning coursework code: behavioral cloning, policy-gradient learning, and actor-critic/DQN methods. The central runtime pattern is an RL trainer configured with an agent, which gathers experience, stores rollouts, and trains a policy or critic; HW3 adds frame-based DQN replay and environment monitoring. The README identifies the course but does not document product workflows. The graph therefore uses explicit source excerpts and SOURCE INDEX references for relationships and omits unsampled script wiring.

Main entry points: [`hw1/rob831/infrastructure/replay_buffer.py`](https://github.com/lecar-lab/16831-f26-hw/blob/main/hw1/rob831/infrastructure/replay_buffer.py) (Rollout replay), [`hw3/rob831/infrastructure/monitor.py`](https://github.com/lecar-lab/16831-f26-hw/blob/main/hw3/rob831/infrastructure/monitor.py) (Environment monitor), [`hw1/rob831/infrastructure/rl_trainer.py`](https://github.com/lecar-lab/16831-f26-hw/blob/main/hw1/rob831/infrastructure/rl_trainer.py) (HW1 trainer), [`hw3/rob831/agents/dqn_agent.py`](https://github.com/lecar-lab/16831-f26-hw/blob/main/hw3/rob831/agents/dqn_agent.py) (DQN agent)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_training["Training loops"]
  node_hw1trainer["HW1 trainer<br/>[rl_trainer.py]"]
  node_hw2trainer["HW2 trainer<br/>[rl_trainer.py]"]
end

subgraph group_agents["Learning agents"]
  node_bcagent["Behavior cloning<br/>[bc_agent.py]"]
  node_pgagent["Policy gradient<br/>[pg_agent.py]"]
  node_acagent["Actor critic<br/>[ac_agent.py]"]
  node_dqnagent["DQN agent<br/>[dqn_agent.py]"]
end

subgraph group_models["Policies and critics"]
  node_mlppolicy["MLP policy<br/>[MLP_policy.py]"]
  node_continuouscritic["Continuous critic"]
  node_dqncritic["DQN critic<br/>[dqn_critic.py]"]
  node_argmax["Argmax policy<br/>[argmax_policy.py]"]
end

subgraph group_runtime["Experience and monitoring"]
  node_replay[("Rollout replay<br/>[replay_buffer.py]")]
  node_rolloututils["Trajectory utilities<br/>[utils.py]"]
  node_torchutil["PyTorch utilities<br/>[pytorch_util.py]"]
  node_logger["Training logger<br/>[logger.py]"]
  node_dqnutils["DQN replay utilities<br/>[dqn_utils.py]"]
  node_monitor["Environment monitor<br/>[monitor.py]"]
  node_stats["Episode statistics<br/>[stats_recorder.py]"]
  node_video["Video recorder<br/>[video_recorder.py]"]
  node_atomic["Atomic file writes<br/>[atomic_write.py]"]
  node_jsonutils["JSON encoding<br/>[json_utils.py]"]
end

node_hw1trainer -->|"creates"| node_logger
node_hw1trainer -->|"instantiates configured agent"| node_bcagent
node_hw2trainer -->|"instantiates configured agent"| node_pgagent
node_hw1trainer -->|"adds collected rollouts"| node_replay
node_hw2trainer -->|"adds collected rollouts"| node_replay
node_bcagent -->|"uses"| node_replay
node_bcagent -->|"uses"| node_mlppolicy
node_pgagent -->|"creates actor"| node_mlppolicy
node_pgagent -->|"creates buffer"| node_replay
node_acagent -->|"uses critic"| node_continuouscritic
node_dqnagent -->|"creates critic"| node_dqncritic
node_dqnagent -->|"creates policy"| node_argmax
node_dqnagent -->|"uses replay and schedule"| node_dqnutils
node_mlppolicy -->|"builds networks with"| node_torchutil
node_replay -->|"converts rollouts with"| node_rolloututils
node_dqncritic -->|"uses tensor utilities"| node_torchutil
node_monitor -->|"uses"| node_stats
node_monitor -->|"uses"| node_video
node_monitor -->|"uses"| node_atomic
node_monitor -->|"encodes with"| node_jsonutils
node_stats -->|"writes through"| node_atomic
node_stats -->|"encodes with"| node_jsonutils

click node_hw1trainer "https://github.com/lecar-lab/16831-f26-hw/blob/main/hw1/rob831/infrastructure/rl_trainer.py"
click node_hw2trainer "https://github.com/lecar-lab/16831-f26-hw/blob/main/hw2/rob831/infrastructure/rl_trainer.py"
click node_bcagent "https://github.com/lecar-lab/16831-f26-hw/blob/main/hw1/rob831/agents/bc_agent.py"
click node_pgagent "https://github.com/lecar-lab/16831-f26-hw/blob/main/hw2/rob831/agents/pg_agent.py"
click node_acagent "https://github.com/lecar-lab/16831-f26-hw/blob/main/hw3/rob831/agents/ac_agent.py"
click node_dqnagent "https://github.com/lecar-lab/16831-f26-hw/blob/main/hw3/rob831/agents/dqn_agent.py"
click node_mlppolicy "https://github.com/lecar-lab/16831-f26-hw/blob/main/hw1/rob831/policies/MLP_policy.py"
click node_replay "https://github.com/lecar-lab/16831-f26-hw/blob/main/hw1/rob831/infrastructure/replay_buffer.py"
click node_rolloututils "https://github.com/lecar-lab/16831-f26-hw/blob/main/hw1/rob831/infrastructure/utils.py"
click node_torchutil "https://github.com/lecar-lab/16831-f26-hw/blob/main/hw1/rob831/infrastructure/pytorch_util.py"
click node_logger "https://github.com/lecar-lab/16831-f26-hw/blob/main/hw1/rob831/infrastructure/logger.py"
click node_continuouscritic "https://github.com/lecar-lab/16831-f26-hw/blob/main/hw3/rob831/critics/bootstrapped_continuous_critic.py"
click node_dqncritic "https://github.com/lecar-lab/16831-f26-hw/blob/main/hw3/rob831/critics/dqn_critic.py"
click node_argmax "https://github.com/lecar-lab/16831-f26-hw/blob/main/hw3/rob831/policies/argmax_policy.py"
click node_dqnutils "https://github.com/lecar-lab/16831-f26-hw/blob/main/hw3/rob831/infrastructure/dqn_utils.py"
click node_monitor "https://github.com/lecar-lab/16831-f26-hw/blob/main/hw3/rob831/infrastructure/monitor.py"
click node_stats "https://github.com/lecar-lab/16831-f26-hw/blob/main/hw3/rob831/infrastructure/monitoring/stats_recorder.py"
click node_video "https://github.com/lecar-lab/16831-f26-hw/blob/main/hw3/rob831/infrastructure/monitoring/video_recorder.py"
click node_atomic "https://github.com/lecar-lab/16831-f26-hw/blob/main/hw3/rob831/infrastructure/monitoring/utils/atomic_write.py"
click node_jsonutils "https://github.com/lecar-lab/16831-f26-hw/blob/main/hw3/rob831/infrastructure/monitoring/utils/json_utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_hw1trainer,node_hw2trainer toneBlue
class node_bcagent,node_pgagent,node_acagent,node_dqnagent toneAmber
class node_mlppolicy,node_continuouscritic,node_dqncritic,node_argmax toneMint
class node_replay,node_rolloututils,node_torchutil,node_logger,node_dqnutils,node_monitor,node_stats,node_video,node_atomic,node_jsonutils toneRose
```

## Components

### Training loops

- [HW1 trainer](https://github.com/lecar-lab/16831-f26-hw/blob/main/hw1/rob831/infrastructure/rl_trainer.py): `hw1/rob831/infrastructure/rl_trainer.py`
- [HW2 trainer](https://github.com/lecar-lab/16831-f26-hw/blob/main/hw2/rob831/infrastructure/rl_trainer.py): `hw2/rob831/infrastructure/rl_trainer.py`

### Learning agents

- [Behavior cloning](https://github.com/lecar-lab/16831-f26-hw/blob/main/hw1/rob831/agents/bc_agent.py): `hw1/rob831/agents/bc_agent.py`
- [Policy gradient](https://github.com/lecar-lab/16831-f26-hw/blob/main/hw2/rob831/agents/pg_agent.py): `hw2/rob831/agents/pg_agent.py`
- [Actor critic](https://github.com/lecar-lab/16831-f26-hw/blob/main/hw3/rob831/agents/ac_agent.py): `hw3/rob831/agents/ac_agent.py`
- [DQN agent](https://github.com/lecar-lab/16831-f26-hw/blob/main/hw3/rob831/agents/dqn_agent.py): `hw3/rob831/agents/dqn_agent.py`

### Policies and critics

- [MLP policy](https://github.com/lecar-lab/16831-f26-hw/blob/main/hw1/rob831/policies/MLP_policy.py): `hw1/rob831/policies/MLP_policy.py`
- [Continuous critic](https://github.com/lecar-lab/16831-f26-hw/blob/main/hw3/rob831/critics/bootstrapped_continuous_critic.py): `hw3/rob831/critics/bootstrapped_continuous_critic.py`
- [DQN critic](https://github.com/lecar-lab/16831-f26-hw/blob/main/hw3/rob831/critics/dqn_critic.py): `hw3/rob831/critics/dqn_critic.py`
- [Argmax policy](https://github.com/lecar-lab/16831-f26-hw/blob/main/hw3/rob831/policies/argmax_policy.py): `hw3/rob831/policies/argmax_policy.py`

### Experience and monitoring

- [Rollout replay](https://github.com/lecar-lab/16831-f26-hw/blob/main/hw1/rob831/infrastructure/replay_buffer.py): `hw1/rob831/infrastructure/replay_buffer.py`
- [Trajectory utilities](https://github.com/lecar-lab/16831-f26-hw/blob/main/hw1/rob831/infrastructure/utils.py): `hw1/rob831/infrastructure/utils.py`
- [PyTorch utilities](https://github.com/lecar-lab/16831-f26-hw/blob/main/hw1/rob831/infrastructure/pytorch_util.py): `hw1/rob831/infrastructure/pytorch_util.py`
- [Training logger](https://github.com/lecar-lab/16831-f26-hw/blob/main/hw1/rob831/infrastructure/logger.py): `hw1/rob831/infrastructure/logger.py`
- [DQN replay utilities](https://github.com/lecar-lab/16831-f26-hw/blob/main/hw3/rob831/infrastructure/dqn_utils.py): `hw3/rob831/infrastructure/dqn_utils.py`
- [Environment monitor](https://github.com/lecar-lab/16831-f26-hw/blob/main/hw3/rob831/infrastructure/monitor.py): `hw3/rob831/infrastructure/monitor.py`
- [Episode statistics](https://github.com/lecar-lab/16831-f26-hw/blob/main/hw3/rob831/infrastructure/monitoring/stats_recorder.py): `hw3/rob831/infrastructure/monitoring/stats_recorder.py`
- [Video recorder](https://github.com/lecar-lab/16831-f26-hw/blob/main/hw3/rob831/infrastructure/monitoring/video_recorder.py): `hw3/rob831/infrastructure/monitoring/video_recorder.py`
- [Atomic file writes](https://github.com/lecar-lab/16831-f26-hw/blob/main/hw3/rob831/infrastructure/monitoring/utils/atomic_write.py): `hw3/rob831/infrastructure/monitoring/utils/atomic_write.py`
- [JSON encoding](https://github.com/lecar-lab/16831-f26-hw/blob/main/hw3/rob831/infrastructure/monitoring/utils/json_utils.py): `hw3/rob831/infrastructure/monitoring/utils/json_utils.py`

## Connections

- HW1 trainer → Training logger: creates (evidence: `hw1/rob831/infrastructure/rl_trainer.py`)
- HW1 trainer → Behavior cloning: instantiates configured agent (evidence: `hw1/rob831/infrastructure/rl_trainer.py`)
- HW2 trainer → Policy gradient: instantiates configured agent (evidence: `hw2/rob831/infrastructure/rl_trainer.py`)
- HW1 trainer → Rollout replay: adds collected rollouts (evidence: `hw1/rob831/infrastructure/rl_trainer.py`)
- HW2 trainer → Rollout replay: adds collected rollouts (evidence: `hw2/rob831/infrastructure/rl_trainer.py`)
- Behavior cloning → Rollout replay: uses (evidence: `hw1/rob831/agents/bc_agent.py`)
- Behavior cloning → MLP policy: uses (evidence: `hw1/rob831/agents/bc_agent.py`)
- Policy gradient → MLP policy: creates actor (evidence: `hw2/rob831/agents/pg_agent.py`)
- Policy gradient → Rollout replay: creates buffer (evidence: `hw2/rob831/agents/pg_agent.py`)
- Actor critic → Continuous critic: uses critic (evidence: `hw3/rob831/agents/ac_agent.py`)
- DQN agent → DQN critic: creates critic (evidence: `hw3/rob831/agents/dqn_agent.py`)
- DQN agent → Argmax policy: creates policy (evidence: `hw3/rob831/agents/dqn_agent.py`)
- DQN agent → DQN replay utilities: uses replay and schedule (evidence: `hw3/rob831/agents/dqn_agent.py`)
- MLP policy → PyTorch utilities: builds networks with (evidence: `hw1/rob831/policies/MLP_policy.py`)
- Rollout replay → Trajectory utilities: converts rollouts with (evidence: `hw1/rob831/infrastructure/replay_buffer.py`)
- DQN critic → PyTorch utilities: uses tensor utilities (evidence: `hw3/rob831/critics/dqn_critic.py`)
- Environment monitor → Episode statistics: uses (evidence: `hw3/rob831/infrastructure/monitor.py`)
- Environment monitor → Video recorder: uses (evidence: `hw3/rob831/infrastructure/monitor.py`)
- Environment monitor → Atomic file writes: uses (evidence: `hw3/rob831/infrastructure/monitor.py`)
- Environment monitor → JSON encoding: encodes with (evidence: `hw3/rob831/infrastructure/monitor.py`)
- Episode statistics → Atomic file writes: writes through (evidence: `hw3/rob831/infrastructure/monitoring/stats_recorder.py`)
- Episode statistics → JSON encoding: encodes with (evidence: `hw3/rob831/infrastructure/monitoring/stats_recorder.py`)
