How jimmyyhwu/tidybot works
Project Type and Purpose
• This is a distributed robotics system (“TidyBot”) combining:
• A GPU‐equipped server (“Server”) running perception, planning, and language modules.
• A mobile robot (“Robot”) running low‐level control and camera capture.
• An offline benchmark suite for evaluating LLM‐based preference summarization.
File‐Structure Analysis
• Top level:
• server/ – all server‐side code (Python scripts for camera server, marker detector, object detector, controller server, teleop/demo interfaces, utilities).
• robot/ – robot‐side code (camera client, controller client, driver interfaces, camera parameters).
• benchmark/ – offline evaluation code + data + notebooks.
• stl/ – 3D parts (mechanical only).
• Notice microservice style: separate servers for cameras, marker detection, object detection, controller, each on its own port. Clients in server/ talk to these.
• Scenarios & preferences live in YAML under server/scenarios and server/preferences.
README Insights
• Two runtimes: Teleoperation mode (teleop.py + controller.py) and Fully Autonomous mode (demo.py + object detector server + controller server).
• Multi‐environment (Conda): tidybot for server & robot, vild for object detector.
• Relies on LLM summarization (OpenAI calls, cached under benchmark/data/openai-cache.pkl, summarization notebooks).
Main Components
A. Server (GPU workstation)
– Camera Server Top (port 6000)
– Camera Server Bottom (6001)
– Marker Detector Server (6002)
– Object Detector Server (6003)
– Controller Server (6004–6006 for each robot)
– Teleop & Demo UI (teleop.py / demo.py)
– Summarization Module (LLM calls + preferences/*.yml)
– Path Planner (shortest_paths module + occupancy_map.py)
– Scenario Manager (loads server/scenarios/*.yml)
– Utility Scripts (calibration, alignment, occupancy map builder)
B. Robot (Mobile base computer)
– Camera Client (camera.py)
– Controller Client (controller_client.py)
– Kinova Arm Interface (kinova.py)
– Motor Drivers (driver/)
– Camera Parameters (camera_params/*.yml)
– Constants & Utilities
C. Hardware & External Services
– Overhead Cameras (physically connected to server)
– Robot Base + Arm
– OpenAI (or another LLM) API endpoint
D. Benchmark Suite (offline)
– benchmark/benchmark.py + notebooks
– LLM cache + method modules
Interactions and Data Flows
• Overhead Cameras → Camera Servers → Server processing
• Server: image → marker & object detector → 2D/3D pose estimates
• Server: pose + scenario + user preferences (from LLM summarization) → Path Planner → motion plan
• Server Controller Server → Robot Controller Client → Robot Driver → Hardware
• Teleop UI → Controller Server → Robot (bypassing path planner)
• Demo UI → entire autonomous pipeline above
• Benchmark suite (offline) calls LLM summarization on example interactions → writes preferences/*.yml
Architectural Patterns & Principles
• Microservices: each processing step (camera, detection, control) runs as an independent server.
• Client–Server: Python clients in server/ invoke remote servers on defined ports.
• Separation of Concerns: perception, planning, control, and UI are distinct layers.
• Config as Code: scenarios & preferences in YAML; camera parameters in YAML/JSON.
• Extensibility: new detectors or planners can be added as new servers.