How zhouzypaul/auto_eval works
Project Type and Purpose
• AutoEval is a full-stack, distributed robotics evaluation framework.
• It orchestrates policy evaluation jobs on real WidowX robots (via ROS + manipulator_gym)
• Provides both CLI (run_eval.py, scripts/) and Web UI (FastAPI + static HTML) for job submission and monitoring
• Supports local vs. remote policy execution (via REST policy servers)
• Integrates success detectors, Slack notifications, visualization, and data-collection pipelines.
File Structure
• Top-level scripts:
• run_eval.py – main CLI orchestrator for evaluation runs
• job_scheduler.py – FastAPI backend for the Web UI
• scripts/ – helper scripts (teleop data collection, classifier fine-tuning, shell launchers)
• Core package: auto_eval/
• policy_server/ – microservice templates & implementations (OpenVLA, Octo, SuSIE, PiZero)
• robot/ – robot environment wrappers, policy clients, command & status utilities
• success_detector/ – VLM-based detectors (Paligemma, human fallback)
• utils/ – Slack bot, timers, info utilities
• visualization/ – local window vs. web-viewer modules
• web_ui/ – server-side helpers for robot control (offline/online toggles)
• Web assets: static/ (HTML/JS for job submission and live view)
• Config & CI: .github/, .flake8, pre-commit, requirements.txt, setup.py
README for Insights
• Architecture callouts:
• Robot server (ROS + manipulator_gym) on lightweight machine
• Orchestrator uses agentlace to communicate with policy & env
• Policy servers expose a REST API accepting observations and returning actions
• FastAPI‐based job scheduler + static front-end for remote submissions
• Slack integration for human-in-the-loop alerts
• Success detection via fine-tuned VLM on Huggingface models
Main Components and Their Relationships
A. Robot Environment Server (ROS + manipulator_gym)
– Deployed on a robot host (e.g. Intel NUC)
– Exposes a gym-style interface via agentlace for step/reset
B. Policy Servers (OpenVLA, Octo, SuSIE, PiZero, plus template)
– Independent microservices (Python REST servers)
– Receive images + proprioceptive data → return 7-dim action vectors
C. Orchestrator / Evaluation Engine
– run_eval.py invokes robot env and policy clients (local or remote)
– Streams observations to success_detector, logs to robot_eval_logger
– Sends Slack messages via auto_eval/utils/slack_bot.py on exceptions or human eval prompts
D. Success Detector Service
– In-process or remote classifier that answers “success?” questions
– Fine-tuned Paligemma VLM, plus human fallback
E. Data Collection & Training Pipelines
– Teleop script (scripts/teleop.py) for collecting demos/images
– Finetuning scripts (scripts/ft_paligemma.py, scripts/ft_openvla.py)
– Image‐filtering & relabeling (scripts/filter_images.py, scripts/relabel_images.py)
F. Web UI & Job Scheduler
– FastAPI app (job_scheduler.py) serves endpoints for job submission and robot status
– Front-end (static/index.html, static/output.html) connects via REST/WebSocket for live video and status
– auto_eval/web_ui/robot_control.py CLI to toggle robot availability
G. Visualization
– Local: auto_eval/visualization/local.py opens a window for live robot feed
– Web: auto_eval/visualization/web_viewer.py pushes frames to browser
Architectural Patterns & Design Principles
• Microservices: each policy server is its own REST service
• Layered / modular design: separation between robot env, policy logic, evaluation orchestration, and UI
• Plugin‐style extensibility: add new policies by dropping modules into policy_server/ or robot/policy.py
• Event-driven notifications via Slack bot
• CLI + Web UI dual interface