How ucd-193ab-ws24/minecapstone works
Project Type and Purpose
• This is a full-stack benchmarking platform for embodied LLM agents, implemented as a 3D sandbox game in Godot (GDScript + C#) with a Python backend that manages LLM-driven agent control via WebSockets.
• Purpose: Generate scripted “goals,” send them to an LLM via a prompt architecture, receive a sequence of high-level actions, and execute them in the Godot environment to quantitatively benchmark agent performance.
Key Components (to include
a. Godot Game Environment
• Scene Graph: world/world.tscn, prefabs/player/agent.tscn
• Autoload Singletons: globals/agent_controller.gd, globals/message_broker.gd, globals/agent_manager.gd
b. Agent Node
• agent.gd (prefabs/player/agent.gd) handles action dispatch and receives commands
c. Prompting Architecture (Python)
• prompts.py: builds the goal-script prompt based on scenario definitions
• llm_service_factory.py + service adapters (openai_service_adapter.py, gemini_service_adapter.py): select and invoke the LLM
• llm_service.py: common interface for sending prompts, receiving responses
d. WebSocket Server
• python/websocket.py: listens for JSON RPC from Godot, routes requests to the LLM service, and pushes back agent action sequences
e. Available Agent Actions
• High-level actions defined in Godot (move_to, attack, pick_up, drop, build, etc.), encoded by the prompt and executed by agent_controller.gd
f. Environment Feedback
• State snapshots (e.g., agent position, world state) sent by Godot to backend to inform chain-of-thought or infinite decision-making
Relationships & Data Flow
1. Godot Agent Initialization
• Agent node starts scenario via ScenarioManager → sets a “goal” (e.g., collect wood).
2. Goal → Prompt Construction
• AgentController calls WebSocket client (godot/websocket.gd) with a “generate_prompt” RPC, including goal and current state.
3. Backend Prompting
• websocket.py receives RPC → prompts.py assembles the goal-script prompt → llm_service_factory picks adapter → sends to LLM API → obtains action list.
4. Action Sequence → Godot
• Backend returns JSON with ordered actions → websocket.gd in Godot parses and forwards to agent_controller.gd.
5. Action Execution
• agent_controller.gd iterates actions → invokes Agent methods (move_to, attack, pick_up, etc.) → environment updates (navigation, physics, animations).
6. Loop / Feedback
• After each action or at checkpoints, Godot can send updated state back to backend for further reasoning or next prompt (infinite decision-making).
Architectural Patterns & Principles
• Client-Server: Godot (client) ↔ Python (server) over WebSocket.
• Autoload Singletons (Godot): centralize agent management and messaging.
• Strategy Pattern: llm_service_factory selects between multiple LLM adapters.
• Modular Scenes: Scenarios, Prefabs, World Generation separated in folders.
Focusing on Agent ↔ Goal-Script Prompting Interaction
• Sequence:
[AgentController.gd] → send “generate_goal_script” RPC →
[websocket.py] → prompts.py.build_goal_prompt() →
[LLM Adapter] → LLM API →
[websocket.py] → return action list JSON →
[agent_controller.gd] → execute actions in world.
• Input: Goal definition (from ScenarioManager)
• Processing: Prompt Template + Scenario context
• Output: JSON array of actions.
Final Checklist
• Essential modules: Godot Client, WebSocket Client, Python WebSocket Server, Prompt Engine, LLM Adapter, LLM Services, Game Environment.
• Available agent actions as an action list block.