How basiclab/ttsg works
Main architectural layers and components
• Input & Language Processing:
• A “Text Input” block that represents the user’s natural language prompt.
• An “LLM Integration” component that interfaces with the OpenAI API (as configured via the .env file) to process the text prompt.
• Scene Planning & Graph Generation:
• A “Scene Planner” module that uses route planning, graph generation, and possibly caching (see agents/navigation and graph/graph_manager.py) to build the layout for the scene.
• “Prompt Processing” module under prompt/ that validates and formats the text input.
• Simulation & Environment Management:
• A “Carla Simulator Integration” component that covers the interaction with the Carla server (installation instructions in the README, version 0.9.15, and related files in maps/ and carla_runner.py).
• A “World Manager” and “Agent Manager” (directory “manager/”) that control various simulated traffic entities like vehicles, cyclists, pedestrians, and walkers.
• Agent and Behavior Modules:
• “Agents” (agents/ directory) including sub-modules for navigation (basic_agent.py, behavior_agent.py, global_route_planner.py, local_planner.py, etc.) and tools (agents/tools/misc.py) represent decision-making and control for individual simulated actors.
• “SafeBench” related modules (safebench/) incorporate safe RL agents, object detection models, and safe policy implementations that adapt the generated scene for training and evaluation.
• Scenario & Experimentation Management:
• “Scenario Definition & Manager” (safebench/scenario/ directory) covers scenario configuration, route files (XML routes and JSON scenarios), and tools that guide training and evaluation.
• “Diversity Test” functionality (diversity_test/ directory) that stores recorded outputs and test configurations.
• Evaluation, Training & Utilities:
• Scripts for training and evaluation (scripts/run_train.py and scripts/run_eval.py) which trigger SafeBench experiments.
• Utility modules (misc/, prompt/, scene_utils/, safebench/util/) that provide cross-cutting concerns like logging, PID control, metric calculation, and file parsing.
Relationships and interactions
• Data Flow:
• The processed scene description then drives the “Scene Planner” module (which uses navigation controllers, planners, and the cached graph from agents/navigation and graph/graph_utils).
• The “World Manager” uses maps (in maps/) and scenarios to instantiate events in the Carla simulation via the “Carla Integration” module.
• From there, agents (vehicles, pedestrians, etc.) execute behaviors based on configurations defined in safebench/agent/ and scenario files.
• Finally, outputs are recorded and optionally analyzed via diversity tests and evaluation scripts.
• Dependencies:
• The “Prompt Processing” module provides formatted text to the “Text-to-Scene” script.
• The “Scene Manager” interacts with “SafeBench” for both reinforcement learning training and evaluation.
• There is clear separation between simulation control (Carla, maps, manager) and agent behavior (agents, safebench/agent, RL policies, object detection).
Architectural patterns and design principles
• Modular design is evident as each directory handles distinct responsibilities (scene generation, agent behavior, simulation management, scenario definition).
• There is an MVC-like separation where the “view” is represented by rendered simulation outputs (recorded images), the “controller” by the manager modules and planning agents, and the “model” appearing in the safe RL and object detection modules.
• The system uses caching (use-cache flag and pre-built graphs) to optimize simulation planning.
• Repurposing of components is clear: same agents used in both text-to-scene generation and diversity testing, as well as in SafeBench training and evaluation.
Technologies & Frameworks
• Python (with PyTorch, as seen from requirements.txt and safebench/agent/model_ckpt/ directories).
• Carla simulator for environmental simulation.
• OpenAI API / GPT for natural language understanding.
• YAML configuration files (found in safebench/agent/config/ and safebench/scenario/config/) to parameterize various agent and scenario settings.
• Reinforcement learning and object detection frameworks (as seen in safebench/agent/rl/ and safebench/agent/object_detection/).
Final remarks
• Be detailed: capture every essential module from text input processing, scene planning, to simulation management and evaluation workflow.