# zhouzypaul/auto_eval architecture

> Architecture diagram of the GitHub repository zhouzypaul/auto_eval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zhouzypaul/auto_eval
- Repository: https://github.com/zhouzypaul/auto_eval

## Overview

##### Project Type and Purpose
- AutoEval is a full-stack, distributed robotics evaluation framework.
- It orchestrates policy evaluation jobs on real WidowX robots (via ROS + manipulator_gym)
- Provides both CLI (`run_eval.py`, scripts/) and Web UI (FastAPI + static HTML) for job submission and monitoring
- Supports local vs. remote policy execution (via REST policy servers)
- Integrates success detectors, Slack notifications, visualization, and data-collection pipelines.

##### File Structure
- Top-level scripts:
- `run_eval.py` – main CLI orchestrator for evaluation runs
- `job_scheduler.py` – FastAPI backend for the Web UI
- `scripts/` – helper scripts (teleop data collection, classifier fine-tuning, shell launchers)
- Core package: `auto_eval/`
- `policy_server/` – microservice templates & implementations (OpenVLA, Octo, SuSIE, PiZero)
- `robot/` – robot environment wrappers, policy clients, command & status utilities
- `success_detector/` – VLM-based detectors (Paligemma, human fallback)
- `utils/` – Slack bot, timers, info utilities
- `visualization/` – local window vs. web-viewer modules
- `web_ui/` – server-side helpers for robot control (offline/online toggles)
- Web assets: `static/` (HTML/JS for job submission and live view)
- Config & CI: `.github/`, `.flake8`, `pre-commit`, `requirements.txt`, `setup.py`

##### README for Insights
- Architecture callouts:
- Robot server (ROS + manipulator_gym) on lightweight machine
- Orchestrator uses agentlace to communicate with policy & env
- Policy servers expose a REST API accepting observations and returning actions
- FastAPI‐based job scheduler + static front-end for remote submissions
- Slack integration for human-in-the-loop alerts
- Success detection via fine-tuned VLM on Huggingface models

##### Main Components and Their Relationships
A. Robot Environment Server (ROS + manipulator_gym)
– Deployed on a robot host (e.g. Intel NUC)
– Exposes a gym-style interface via agentlace for step/reset
B. Policy Servers (OpenVLA, Octo, SuSIE, PiZero, plus template)
– Independent microservices (Python REST servers)
– Receive images + proprioceptive data → return 7-dim action vectors
C. Orchestrator / Evaluation Engine
– `run_eval.py` invokes robot env and policy clients (local or remote)
– Streams observations to success_detector, logs to robot_eval_logger
– Sends Slack messages via `auto_eval/utils/slack_bot.py` on exceptions or human eval prompts
D. Success Detector Service
– In-process or remote classifier that answers “success?” questions
– Fine-tuned Paligemma VLM, plus human fallback
E. Data Collection & Training Pipelines
– Teleop script (`scripts/teleop.py`) for collecting demos/images
– Finetuning scripts (`scripts/ft_paligemma.py`, `scripts/ft_openvla.py`)
– Image‐filtering & relabeling (`scripts/filter_images.py`, `scripts/relabel_images.py`)
F. Web UI & Job Scheduler
– FastAPI app (`job_scheduler.py`) serves endpoints for job submission and robot status
– Front-end (`static/index.html`, `static/output.html`) connects via REST/WebSocket for live video and status
– `auto_eval/web_ui/robot_control.py` CLI to toggle robot availability
G. Visualization
– Local: `auto_eval/visualization/local.py` opens a window for live robot feed
– Web: `auto_eval/visualization/web_viewer.py` pushes frames to browser

##### Architectural Patterns & Design Principles
- Microservices: each policy server is its own REST service
- Layered / modular design: separation between robot env, policy logic, evaluation orchestration, and UI
- Plugin‐style extensibility: add new policies by dropping modules into `policy_server/` or `robot/policy.py`
- Event-driven notifications via Slack bot
- CLI + Web UI dual interface

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Robot Host
    subgraph "Robot Host" 
        RobotEnv["Robot Environment Server\n(ROS + manipulator_gym)"]:::infrastructure
    end

    %% Policy Servers
    subgraph "Policy Servers" 
        PolicyServers["auto_eval/policy_server"]:::infrastructure
    end

    %% Orchestrator / Evaluation Engine
    subgraph "Orchestrator / Evaluation Engine"
        Orchestrator["Orchestrator Engine\nrun_eval.py"]:::core
        PolicyClients["Policy Clients\nauto_eval/robot/policy_clients.py"]:::core
        SlackBot["Slack Notification Utility"]:::core
        SuccessDetector["Success Detector Service"]:::core
    end

    %% Data Collection & Training Pipelines
    subgraph "Data Collection & Training Pipelines"
        Teleop["Teleop Data Collection\nscripts/teleop.py"]:::pipeline
        FtPaligemma["Fine-tuning Paligemma\nscripts/ft_paligemma.py"]:::pipeline
        FtOpenVLA["Fine-tuning OpenVLA\nscripts/ft_openvla.py"]:::pipeline
        FilterImages["Image Filtering\nscripts/filter_images.py"]:::pipeline
        RelabelImages["Image Relabeling\nscripts/relabel_images.py"]:::pipeline
    end

    %% Web UI & Job Scheduler
    subgraph "Web UI & Job Scheduler"
        JobScheduler["FastAPI Job Scheduler\njob_scheduler.py"]:::ui
        FrontEnd["Front-end Assets\nindex.html / output.html"]:::ui
        RobotControl["Robot Availability CLI\nauto_eval/web_ui/robot_control.py"]:::ui
        Launcher["Web-UI Launcher Helpers\nauto_eval/web_ui/launcher.py"]:::ui
    end

    %% Visualization Clients
    subgraph "Visualization Clients"
        LocalViewer["Local Viewer\nauto_eval/visualization/local.py"]:::ui
        WebViewer["Web Viewer\nauto_eval/visualization/web_viewer.py"]:::ui
    end

    %% External Services
    SlackAPI((Slack API)):::external
    Huggingface((Huggingface VLM)):::external
    Agentlace((agentlace network)):::external
    Browser((Browser Client)):::ui

    %% Connections
    Orchestrator -->|agentlace gRPC| RobotEnv
    Orchestrator -->|HTTP POST observations/actions| PolicyServers
    PolicyServers -->|action vectors| Orchestrator
    Orchestrator -->|"in-process calls"| SuccessDetector
    Orchestrator -->|"HTTPS calls"| SlackAPI
    Orchestrator -->|streams data| LocalViewer
    Orchestrator -->|streams frames| WebViewer

    Browser -->|HTTP/WS| JobScheduler
    JobScheduler -->|serves static| Browser

    Teleop -->|collect demos| PolicyServers
    FtOpenVLA -->|upload models| PolicyServers
    FtPaligemma -->|fine-tune| SuccessDetector
    FilterImages -->|filter data| SuccessDetector
    RelabelImages -->|relabel| SuccessDetector

    SlackAPI -->|alerts| Orchestrator
    Huggingface -->|models| SuccessDetector
    Agentlace -->|interface| RobotEnv

    Orchestrator --> PolicyClients
    PolicyClients --> PolicyServers
    Orchestrator --> SlackBot

    %% Click Events
    click RobotEnv "https://github.com/zhouzypaul/auto_eval/tree/main/auto_eval/robot/"
    click PolicyServers "https://github.com/zhouzypaul/auto_eval/tree/main/auto_eval/policy_server/"
    click Orchestrator "https://github.com/zhouzypaul/auto_eval/blob/main/run_eval.py"
    click PolicyClients "https://github.com/zhouzypaul/auto_eval/blob/main/auto_eval/robot/policy_clients.py"
    click SlackBot "https://github.com/zhouzypaul/auto_eval/blob/main/auto_eval/utils/slack_bot.py"
    click SuccessDetector "https://github.com/zhouzypaul/auto_eval/tree/main/auto_eval/success_detector/"
    click Teleop "https://github.com/zhouzypaul/auto_eval/blob/main/scripts/teleop.py"
    click FtPaligemma "https://github.com/zhouzypaul/auto_eval/blob/main/scripts/ft_paligemma.py"
    click FtOpenVLA "https://github.com/zhouzypaul/auto_eval/blob/main/scripts/ft_openvla.py"
    click FilterImages "https://github.com/zhouzypaul/auto_eval/blob/main/scripts/filter_images.py"
    click RelabelImages "https://github.com/zhouzypaul/auto_eval/blob/main/scripts/relabel_images.py"
    click JobScheduler "https://github.com/zhouzypaul/auto_eval/blob/main/job_scheduler.py"
    click FrontEnd "https://github.com/zhouzypaul/auto_eval/blob/main/static/index.html"
    click RobotControl "https://github.com/zhouzypaul/auto_eval/blob/main/auto_eval/web_ui/robot_control.py"
    click Launcher "https://github.com/zhouzypaul/auto_eval/blob/main/auto_eval/web_ui/launcher.py"
    click LocalViewer "https://github.com/zhouzypaul/auto_eval/blob/main/auto_eval/visualization/local.py"
    click WebViewer "https://github.com/zhouzypaul/auto_eval/blob/main/auto_eval/visualization/web_viewer.py"

    %% Styles
    classDef infrastructure fill:#B3CDE3,stroke:#005B96,color:#03396B
    classDef core fill:#C7E9C0,stroke:#238B45,color:#006D2C
    classDef ui fill:#FED9A6,stroke:#E6550D,color:#A63603
    classDef external fill:#F2F0F7,stroke:#969696,color:#636363
    classDef pipeline fill:#FFF2AE,stroke:#D8B365,color:#8C510A
```
