How nt505-p21-kltn/schedule-agent works
Project Type and Purpose
• This repository implements a deep reinforcement learning framework (using Deep Q-Learning) to optimize the control of a traffic signal at a single four-way intersection.
• It integrates the SUMO traffic simulator as the environment for simulating traffic as well as TensorFlow/Keras for building and training the neural network.
• Key functionalities include simulation control (both training and testing modes), experience replay (memory management), dynamic neural network configuration, and visualizing performance metrics.
Analyzing the File Structure for Architectural Insights
• Top-Level Files: The presence of files like LICENSE, README.md, and the Jupyter notebook (TLCS-colab-training.ipynb) suggest that the repository is meant to be user-friendly and shareable.
• Directory “TLCS”: This is the main module containing all the core classes and scripts.
• Scripts like training_main.py and testing_main.py handle the overall training loop and testing routines.
• The "training_simulation.py" and "testing_simulation.py" files indicate that there are two simulation routines adapted to different phases.
• “generator.py” (within TLCS) implies functionality related to the TrafficGenerator class, which creates the vehicle routes (episode_routes.rou.xml is generated in the "intersection" folder).
• The “model.py” file along with the “models” directory (with subfolders model_5, model_7, etc.) store the definitions of the deep neural network and its trained weights.
• Supplemental files such as memory.py, visualization.py, and utils.py indicate modularized responsibilities:
▪ Memory management (experience replay storage).
▪ Visualization (graph generation, plotting performance metrics).
▪ Utility functions (file handling, model versioning, etc.).
• The “intersection” folder contains SUMO-specific configuration files (environment.net.xml and sumo_config.sumocfg) which define the traffic network and how SUMO ties in with the simulation.
Key Components & Their Interactions (to be reflected
a. Simulation Environment
• SUMO Simulator: Uses the network XML and configuration files to run traffic simulations.
• Environment: Defined by the "intersection" directory, where the SUMO network (environment.net.xml) and configuration (sumo_config.sumocfg) are specified.
b. Traffic Generation Module
• TrafficGenerator class: Responsible for generating vehicle routes represented by episode_routes.rou.xml.
c. Deep Q-Learning Agent
• Model Component:
▪ Neural Network: Constructed dynamically based on parameters from the settings file, implemented with TensorFlow/Keras.
▪ Two-model variations (one for training, one for testing) as defined in model.py.
• Experience Replay:
▪ Memory class: Stores observed state transition samples and supplies randomized batches for training.
d. Simulation Controller
• Simulation Class:
▪ Two variants exist (training_simulation.py and testing_simulation.py) to run the simulation episodes.
▪ Manages interaction with SUMO for state acquisition (e.g., get_state), applying actions (traffic light phase changes), and handling simulation loop.
e. Visualization & Utilities
• Visualization class: Processes and plots data (e.g., wait times, queue lengths).
• Utils functions: Handles tasks like creating new model folders and loading/saving models.
Relationships and Data Flows
• The Simulation Controller initiates an episode by triggering SUMO via the configuration files.
• Each simulation cycle:
• Receives the current state (discretized lanes) from the SUMO environment.
• Feeds this state into the deep neural network (Model component) to predict the optimal traffic light phase (i.e., the action).
• Applies the chosen action, which is executed in the simulation (adjusting traffic signals).
• Collects the resulting reward (cumulative waiting time change) from the environment.
• Passes experience data (state, action, reward, next state) to the Memory module.
• At the end of each simulation episode, the accumulated experiences are sampled for training the deep network.
• Visualization is typically executed post-episode, using stored data (graphs, metrics) to present learned performance progress.
Architectural Patterns and Design Principles
• Modularity: Each major responsibility (simulation, generation, training, memory, visualization) is encapsulated in its own module/class.
• Separation of Concerns:
• Training and testing routines are isolated in separate scripts and settings files.
• Environment interfacing is separate from the learning and memory mechanisms.
• Data Flow/Control Loop: Emphasize the reinforcement learning loop, in which state updates trigger actions, rewards are generated, and learning updates occur in batches via experience replay.
• Configurability: Use of .ini files for both training and testing demonstrates a design that separates configuration from code.