# jimmyyhwu/tidybot architecture

> Architecture diagram of the GitHub repository jimmyyhwu/tidybot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jimmyyhwu/tidybot
- Repository: https://github.com/jimmyyhwu/tidybot

## Overview

##### Project Type and Purpose
- This is a distributed robotics system (“TidyBot”) combining:
- A GPU‐equipped server (“Server”) running perception, planning, and language modules.
- A mobile robot (“Robot”) running low‐level control and camera capture.
- An offline benchmark suite for evaluating LLM‐based preference summarization.

##### File‐Structure Analysis
- Top level:
- `server/` – all server‐side code (Python scripts for camera server, marker detector, object detector, controller server, teleop/demo interfaces, utilities).
- `robot/` – robot‐side code (camera client, controller client, driver interfaces, camera parameters).
- `benchmark/` – offline evaluation code + data + notebooks.
- `stl/` – 3D parts (mechanical only).
- Notice microservice style: separate servers for cameras, marker detection, object detection, controller, each on its own port. Clients in `server/` talk to these.
- Scenarios & preferences live in YAML under `server/scenarios` and `server/preferences`.

##### README Insights
- Two runtimes: Teleoperation mode (`teleop.py` + `controller.py`) and Fully Autonomous mode (`demo.py` + object detector server + controller server).
- Multi‐environment (Conda): `tidybot` for server & robot, `vild` for object detector.
- Relies on LLM summarization (OpenAI calls, cached under `benchmark/data/openai-cache.pkl`, summarization notebooks).

##### Main Components
A. Server (GPU workstation)
– Camera Server Top (port 6000)
– Camera Server Bottom (6001)
– Marker Detector Server (6002)
– Object Detector Server (6003)
– Controller Server (6004–6006 for each robot)
– Teleop & Demo UI (`teleop.py` / `demo.py`)
– Summarization Module (LLM calls + `preferences/*.yml`)
– Path Planner (`shortest_paths` module + `occupancy_map.py`)
– Scenario Manager (loads `server/scenarios/*.yml`)
– Utility Scripts (calibration, alignment, occupancy map builder)

B. Robot (Mobile base computer)
– Camera Client (`camera.py`)
– Controller Client (`controller_client.py`)
– Kinova Arm Interface (`kinova.py`)
– Motor Drivers (`driver/`)
– Camera Parameters (`camera_params/*.yml`)
– Constants & Utilities

C. Hardware & External Services
– Overhead Cameras (physically connected to server)
– Robot Base + Arm
– OpenAI (or another LLM) API endpoint

D. Benchmark Suite (offline)
– `benchmark/benchmark.py` + notebooks
– LLM cache + method modules

##### Interactions and Data Flows
- Overhead Cameras → Camera Servers → Server processing
- Server: image → marker & object detector → 2D/3D pose estimates
- Server: pose + scenario + user preferences (from LLM summarization) → Path Planner → motion plan
- Server Controller Server → Robot Controller Client → Robot Driver → Hardware
- Teleop UI → Controller Server → Robot (bypassing path planner)
- Demo UI → entire autonomous pipeline above
- Benchmark suite (offline) calls LLM summarization on example interactions → writes `preferences/*.yml`

##### Architectural Patterns & Principles
- Microservices: each processing step (camera, detection, control) runs as an independent server.
- Client–Server: Python clients in `server/` invoke remote servers on defined ports.
- Separation of Concerns: perception, planning, control, and UI are distinct layers.
- Config as Code: scenarios & preferences in YAML; camera parameters in YAML/JSON.
- Extensibility: new detectors or planners can be added as new servers.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Cameras"
        Overhead_Top["Overhead Camera Top"]:::hardware
        Overhead_Bottom["Overhead Camera Bottom"]:::hardware
    end

    subgraph "Server (GPU workstation)"
        subgraph "Camera Servers"
            CameraTop["Camera Server Top (6000)"]:::service
            CameraBottom["Camera Server Bottom (6001)"]:::service
        end
        Marker["Marker Detector Server (6002)"]:::service
        ObjectDet["Object Detector Server (6003)"]:::service
        ControllerSrv["Controller Server (6004)"]:::service
        UI_Teleop["Teleoperation UI"]:::ui
        UI_Demo["Autonomous Demo UI"]:::ui
        Summarizer["Summarization Module"]:::external
        PrefStore["Preferences Store"]:::storage
        ScenarioMgr["Scenario Manager"]:::storage
        PathPlanner["Path Planner"]:::service
        OccMap["Occupancy Map Builder"]:::service
        CalibCharuco["Calibrate Charuco Board"]:::utility
        CreateCharuco["Create Charuco Board"]:::utility
        Align["Camera Alignment Utility"]:::utility
        MarkerUtil["Marker Creation Utility"]:::utility
    end

    subgraph "Robot (Mobile Base)"
        CamClient["Camera Client"]:::service
        CtrlClient["Controller Client"]:::service
        Kinova["Kinova Arm Interface"]:::hardware
        Drivers["Motor Drivers"]:::hardware
        CamParams["Camera Parameters"]:::storage
        RobotUtils["Constants & Utilities"]:::utility
    end

    subgraph "External Services"
        OpenAI["OpenAI API"]:::external
    end

    subgraph "Benchmark Suite (Offline)"
        BenchRunner["Benchmark Runner"]:::service
        BenchNotebooks["Benchmark Notebooks"]:::storage
        OpenAICache["OpenAI Cache & Utils"]:::storage
        BenchScen["Benchmark Scenarios"]:::storage
    end

    Overhead_Top --> CameraTop
    Overhead_Bottom --> CameraBottom

    CameraTop --> Marker
    CameraBottom --> Marker
    Marker --> ObjectDet
    ObjectDet --> ControllerSrv
    Marker --> ControllerSrv
    ControllerSrv --> CtrlClient
    CtrlClient --> Drivers
    CtrlClient --> Kinova

    CamClient --> CameraTop
    CamClient --> CameraBottom

    UI_Teleop --> ControllerSrv
    UI_Demo --> ControllerSrv
    UI_Demo --> PathPlanner

    Summarizer --> PrefStore
    ScenarioMgr --> PathPlanner
    Summarizer --> PathPlanner
    Summarizer <--> OpenAI

    PathPlanner --> ControllerSrv
    OccMap --> PathPlanner

    BenchRunner --> OpenAICache
    BenchRunner --> OpenAI
    BenchRunner --> BenchNotebooks
    BenchRunner --> BenchScen

    click CameraTop "https://github.com/jimmyyhwu/tidybot/blob/main/server/camera_server.py"
    click CameraBottom "https://github.com/jimmyyhwu/tidybot/blob/main/server/camera_server.py"
    click Marker "https://github.com/jimmyyhwu/tidybot/blob/main/server/marker_detector_server.py"
    click ObjectDet "https://github.com/jimmyyhwu/tidybot/blob/main/server/object_detector_server.py"
    click ControllerSrv "https://github.com/jimmyyhwu/tidybot/blob/main/server/controller_server.py"
    click UI_Teleop "https://github.com/jimmyyhwu/tidybot/blob/main/server/teleop.py"
    click UI_Demo "https://github.com/jimmyyhwu/tidybot/blob/main/server/demo.py"
    click Summarizer "https://github.com/jimmyyhwu/tidybot/blob/main/server/summarization.ipynb"
    click PrefStore "https://github.com/jimmyyhwu/tidybot/blob/main/server/preferences/*.yml"
    click ScenarioMgr "https://github.com/jimmyyhwu/tidybot/blob/main/server/scenarios/*.yml"
    click PathPlanner "https://github.com/jimmyyhwu/tidybot/blob/main/server/shortest_paths/shortest_paths.pyx"
    click OccMap "https://github.com/jimmyyhwu/tidybot/blob/main/server/occupancy_map.py"
    click CalibCharuco "https://github.com/jimmyyhwu/tidybot/blob/main/server/calibrate_charuco.py"
    click CreateCharuco "https://github.com/jimmyyhwu/tidybot/blob/main/server/create_charuco_board.py"
    click Align "https://github.com/jimmyyhwu/tidybot/blob/main/server/camera_alignment.py"
    click MarkerUtil "https://github.com/jimmyyhwu/tidybot/blob/main/server/create_markers.py"
    click CamClient "https://github.com/jimmyyhwu/tidybot/blob/main/robot/camera.py"
    click CtrlClient "https://github.com/jimmyyhwu/tidybot/blob/main/robot/controller.py"
    click Kinova "https://github.com/jimmyyhwu/tidybot/blob/main/robot/kinova.py"
    click Drivers "https://github.com/jimmyyhwu/tidybot/tree/main/robot/driver/"
    click CamParams "https://github.com/jimmyyhwu/tidybot/blob/main/robot/camera_params/*.yml"
    click RobotUtils "https://github.com/jimmyyhwu/tidybot/blob/main/robot/constants.py"
    click BenchRunner "https://github.com/jimmyyhwu/tidybot/blob/main/benchmark/benchmark.py"
    click BenchNotebooks "https://github.com/jimmyyhwu/tidybot/blob/main/benchmark/*.ipynb"
    click OpenAICache "https://github.com/jimmyyhwu/tidybot/blob/main/benchmark/data/openai-cache.pkl"
    click BenchScen "https://github.com/jimmyyhwu/tidybot/blob/main/benchmark/scenarios.yml"

    classDef service fill:#D6EAF8,stroke:#2980B9;
    classDef hardware fill:#ABEBC6,stroke:#1D8348;
    classDef external fill:#F9E79F,stroke:#B7950B;
    classDef storage fill:#D5DBDB,stroke:#626567;
    classDef ui fill:#FCF3CF,stroke:#B7950B,shape:parallelogram;
    classDef utility fill:#E8DAEF,stroke:#8E44AD;
```
