# nt505-p21-kltn/schedule-agent architecture

> Architecture diagram of the GitHub repository nt505-p21-kltn/schedule-agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nt505-p21-kltn/schedule-agent
- Repository: https://github.com/nt505-p21-kltn/schedule-agent

## Overview

##### Project Type and Purpose
- This repository implements a deep reinforcement learning framework (using Deep Q-Learning) to optimize the control of a traffic signal at a single four-way intersection.
- It integrates the SUMO traffic simulator as the environment for simulating traffic as well as TensorFlow/Keras for building and training the neural network.
- Key functionalities include simulation control (both training and testing modes), experience replay (memory management), dynamic neural network configuration, and visualizing performance metrics.

##### Analyzing the File Structure for Architectural Insights
- Top-Level Files: The presence of files like LICENSE, README.md, and the Jupyter notebook (TLCS-colab-training.ipynb) suggest that the repository is meant to be user-friendly and shareable.
- Directory “TLCS”: This is the main module containing all the core classes and scripts.
- Scripts like training_main.py and testing_main.py handle the overall training loop and testing routines.
- The "training_simulation.py" and "testing_simulation.py" files indicate that there are two simulation routines adapted to different phases.
- “generator.py” (within TLCS) implies functionality related to the TrafficGenerator class, which creates the vehicle routes (episode_routes.rou.xml is generated in the "intersection" folder).
- The “model.py” file along with the “models” directory (with subfolders model_5, model_7, etc.) store the definitions of the deep neural network and its trained weights.
- Supplemental files such as memory.py, visualization.py, and utils.py indicate modularized responsibilities:
▪ Memory management (experience replay storage).
▪ Visualization (graph generation, plotting performance metrics).
▪ Utility functions (file handling, model versioning, etc.).
- The “intersection” folder contains SUMO-specific configuration files (environment.net.xml and sumo_config.sumocfg) which define the traffic network and how SUMO ties in with the simulation.

##### Key Components & Their Interactions (to be reflected
a. Simulation Environment
- SUMO Simulator: Uses the network XML and configuration files to run traffic simulations.
- Environment: Defined by the "intersection" directory, where the SUMO network (environment.net.xml) and configuration (sumo_config.sumocfg) are specified.
b. Traffic Generation Module
- TrafficGenerator class: Responsible for generating vehicle routes represented by episode_routes.rou.xml.
c. Deep Q-Learning Agent
- Model Component:
▪ Neural Network: Constructed dynamically based on parameters from the settings file, implemented with TensorFlow/Keras.
▪ Two-model variations (one for training, one for testing) as defined in model.py.
- Experience Replay:
▪ Memory class: Stores observed state transition samples and supplies randomized batches for training.
d. Simulation Controller
- Simulation Class:
▪ Two variants exist (training_simulation.py and testing_simulation.py) to run the simulation episodes.
▪ Manages interaction with SUMO for state acquisition (e.g., get_state), applying actions (traffic light phase changes), and handling simulation loop.
e. Visualization & Utilities
- Visualization class: Processes and plots data (e.g., wait times, queue lengths).
- Utils functions: Handles tasks like creating new model folders and loading/saving models.

##### Relationships and Data Flows
- The Simulation Controller initiates an episode by triggering SUMO via the configuration files.
- Each simulation cycle:
- Receives the current state (discretized lanes) from the SUMO environment.
- Feeds this state into the deep neural network (Model component) to predict the optimal traffic light phase (i.e., the action).
- Applies the chosen action, which is executed in the simulation (adjusting traffic signals).
- Collects the resulting reward (cumulative waiting time change) from the environment.
- Passes experience data (state, action, reward, next state) to the Memory module.
- At the end of each simulation episode, the accumulated experiences are sampled for training the deep network.
- Visualization is typically executed post-episode, using stored data (graphs, metrics) to present learned performance progress.

##### Architectural Patterns and Design Principles
- Modularity: Each major responsibility (simulation, generation, training, memory, visualization) is encapsulated in its own module/class.
- Separation of Concerns:
- Training and testing routines are isolated in separate scripts and settings files.
- Environment interfacing is separate from the learning and memory mechanisms.
- Data Flow/Control Loop: Emphasize the reinforcement learning loop, in which state updates trigger actions, rewards are generated, and learning updates occur in batches via experience replay.
- Configurability: Use of .ini files for both training and testing demonstrates a design that separates configuration from code.

## Diagram (Mermaid)

```mermaid
graph TD
    subgraph "External Dependencies"
        TF["TensorFlow/Keras"]:::external
    end

    subgraph "Environment"
        SUMO["SUMO Simulator Environment"]:::sims
    end

    subgraph "Traffic Generation"
        TG["Traffic Generator"]:::tg
    end

    subgraph "Simulation Controllers"
        SimTrain["Simulation Controller (Training)"]:::sim
        SimTest["Simulation Controller (Testing)"]:::sim
    end

    subgraph "Deep Q-Learning Agent"
        NN["Neural Network Model"]:::nn
        Memory["Experience Replay (Memory)"]:::memory
    end

    subgraph "Supporting Modules"
        Viz["Visualization Module"]:::viz
        Utils["Utility Functions"]:::utils
    end

    %% Data Flow Connections
    TG -->|"generates_routes"| SUMO
    SUMO -->|"state_update"| SimTrain
    SUMO -->|"state_update"| SimTest
    SimTrain -->|"state_to_agent"| NN
    SimTest -->|"state_to_agent"| NN
    NN -->|"action_command"| SimTrain
    NN -->|"action_command"| SimTest
    SUMO -->|"reward_feedback"| SimTrain
    SUMO -->|"reward_feedback"| SimTest
    SimTrain -->|"experience_capture"| Memory
    SimTest -->|"experience_capture"| Memory
    Memory -->|"experience_batch"| NN
    NN -->|"decision_loop"| Memory
    SimTrain -->|"metrics_data"| Viz
    SimTest -->|"metrics_data"| Viz
    Utils -->|"config_update"| NN
    Utils -->|"config_update"| SimTrain
    Utils -->|"config_update"| SimTest
    TF -->|"framework_usage"| NN

    %% Click Events
    click SUMO "https://github.com/nt505-p21-kltn/schedule-agent/tree/main/TLCS/intersection"
    click TG "https://github.com/nt505-p21-kltn/schedule-agent/blob/main/TLCS/generator.py"
    click NN "https://github.com/nt505-p21-kltn/schedule-agent/blob/main/TLCS/model.py"
    click NN "https://github.com/nt505-p21-kltn/schedule-agent/tree/main/TLCS/models"
    click Memory "https://github.com/nt505-p21-kltn/schedule-agent/blob/main/TLCS/memory.py"
    click SimTrain "https://github.com/nt505-p21-kltn/schedule-agent/blob/main/TLCS/training_main.py"
    click SimTrain "https://github.com/nt505-p21-kltn/schedule-agent/blob/main/TLCS/training_simulation.py"
    click SimTest "https://github.com/nt505-p21-kltn/schedule-agent/blob/main/TLCS/testing_main.py"
    click SimTest "https://github.com/nt505-p21-kltn/schedule-agent/blob/main/TLCS/testing_simulation.py"
    click Viz "https://github.com/nt505-p21-kltn/schedule-agent/blob/main/TLCS/visualization.py"
    click Utils "https://github.com/nt505-p21-kltn/schedule-agent/blob/main/TLCS/utils.py"

    %% Styles
    classDef external fill:#f9e79f,stroke:#D35400,stroke-width:2px;
    classDef sims fill:#bbdef0,stroke:#0D47A1,stroke-width:2px;
    classDef tg fill:#c8e6c9,stroke:#2E7D32,stroke-width:2px;
    classDef sim fill:#fff9c4,stroke:#F57F17,stroke-width:2px;
    classDef nn fill:#f8bbd0,stroke:#AD1457,stroke-width:2px;
    classDef memory fill:#ffcdd2,stroke:#C62828,stroke-width:2px;
    classDef viz fill:#b3e5fc,stroke:#0277BD,stroke-width:2px;
    classDef utils fill:#ead7f2,stroke:#6A1B9A,stroke-width:2px;
```
