How internrobotics/genmanip-client works
1. Package shape and entrypoint
- This is a Python CLI package, installed from pyproject.toml and exposed through the gmp command.
- src/genmanip_client/__main__.py and cli.py act as the top-level dispatch layer, routing subcommands to the feature-specific modules.
2. Command-oriented architecture
- The repo is organized around separate CLI commands rather than a single monolithic app.
- Each major user workflow has its own pair of modules: a *_cli.py interface layer and a corresponding client/helper module where the actual work happens.
3. Core evaluation workflow
- eval_cli.py and eval_client.py implement the local evaluation runner.
- This client connects to the GenManip eval server, runs episodes using fake actions or a provided online endpoint, and can optionally trigger plot generation after each completed episode.
4. Job submission and status management
- submit.py / submit_cli.py handle sending evaluation jobs to the server.
- status_cli.py provides a read-only status path for polling server-side job state.
- These commands represent the main network boundary to the GenManip evaluation backend.
5. Online evaluation extension
- extensions/online_client.py and extensions/online_cli.py support the “online” workflow where a local model connects to a remote evaluation service.
- The flow is: create a task on the online service, poll until an endpoint is assigned, then point gmp eval at that endpoint.
- This is a separate external service boundary from the standard eval server.
6. Leaderboard integration
- extensions/leaderboard.py and extensions/leaderboard_cli.py provide listing/submission operations for benchmark leaderboard usage.
- This subsystem bridges local run artifacts with a server-side leaderboard endpoint, using run IDs and benchmark labels as the key identifiers.
7. Visualization subsystem
- extensions/visualize.py and extensions/visualize_cli.py implement a local results browser.
- It serves a local HTTPS web server that lets a user inspect runs, task-level success rates, and replay episodes in the Rerun viewer.
- This is the most UI-heavy part of the repo, but it is still a CLI-backed local service rather than a standalone web app.
8. Episode-level data flow
- The visualizer reads saved evaluation outputs from the filesystem, especially episode directories under saved/eval_results or a user-supplied project root.
- It builds per-episode .rrd cache files inside each episode directory as .genmanip_vis.rrd, then loads those recordings into the embedded Rerun viewer.
9. Plotting and post-processing
- plot_cli.py and vis_utils.py handle offline episode post-processing.
- They consume steps.jsonl plus recorded video files like merged*.mp4, then generate action_plot.png, state_plot.png, and a merged video with plots overlaid.
- ffmpeg is an optional external tool that improves the final video encoding.
10. Cleanup and workspace hygiene
- clean_cli.py is a filesystem maintenance command for removing generated artifacts rather than user-authored content.
- It targets cached mesh data, evaluation results, logs, and recursive lock/tmp leftovers, while intentionally preserving user-facing folders like client_results, saved/tasks, and saved/demonstrations.
11. Filesystem as the main local boundary
- A large part of the architecture is file-based: eval runs write episode artifacts, the plotter reads them back, and the visualizer caches derived .rrd files next to the raw episode data.
- This means the repository’s “state” is split between transient server interactions and durable on-disk run artifacts.
12. Optional dependencies and runtime constraints
- The repo supports optional extras for decoding tensors/images/numpy arrays and for visualization via rerun-sdk.
- The visualizer path depends on Python 3.11+ for the pinned Rerun stack, and it serves content over HTTPS because browser-side playback uses WebCodecs.
13. Test coverage focus
- The existing tests focus on the cleanup path and the online client boundary.
- That suggests the most important correctness risks are in filesystem mutation and remote task/endpoint polling rather than in any pure computation layer.
14. How to draw the architecture
- A good diagram would center on the CLI dispatcher, then fan out into five backend paths: submit/status, eval client, online evaluation, plotting, and visualization/leaderboard.
- Each path either talks to the GenManip server(s) over HTTP or reads/writes run artifacts on disk, with the visualizer adding a local HTTPS server and browser-facing Rerun viewer on top.
Main entry points: cli.py Dispatcher · eval_client.py Eval client · visualize.py Visualizer · online_client.py Online client
Components 16
Core CLI
Entrypoint, command dispatch, and the main evaluation/job flows.
- gmp entry · Installs the gmp command and starts CLI dispatch.
src/genmanip_client/__main__.py - Dispatcher · Routes subcommands to the feature-specific CLI modules.
src/genmanip_client/cli.py - Eval CLI · User-facing command interface for running evaluations.
src/genmanip_client/eval_cli.py - Eval client · Connects to the eval server, runs episodes, and can trigger post-episode plots.
src/genmanip_client/eval_client.py - Submit CLI · Command interface for submitting evaluation jobs.
src/genmanip_client/submit_cli.py - Submit logic · Builds and sends job submission requests to the evaluation backend.
src/genmanip_client/submit.py - Status CLI · Read-only command for polling server-side job state.
src/genmanip_client/status_cli.py - Plot CLI · Generates plots and merged episode media from recorded run artifacts.
src/genmanip_client/plot_cli.py - Plot utils · Reads steps and video artifacts, then writes action/state plots and overlaid video outputs.
src/genmanip_client/vis_utils.py - Cleanup · Removes generated caches, eval results, logs, and temporary leftovers.
src/genmanip_client/clean_cli.py
Extensions
Optional workflows for online eval, leaderboard access, and visualization.
- Online CLI · Command interface for the online evaluation workflow.
src/genmanip_client/extensions/online_cli.py - Online client · Creates online tasks, polls for an assigned endpoint, and bridges local eval to the remote service.
src/genmanip_client/extensions/online_client.py - Leaderboard CLI · Command interface for leaderboard listing and submission operations.
src/genmanip_client/extensions/leaderboard_cli.py - Leaderboard · Maps local run artifacts and benchmark labels to the leaderboard endpoint.
src/genmanip_client/extensions/leaderboard.py - Viz CLI · Command interface for the local results browser and Rerun-backed viewer.
src/genmanip_client/extensions/visualize_cli.py - Visualizer · Serves HTTPS UI, reads saved episode artifacts, and builds per-episode Rerun caches.
src/genmanip_client/extensions/visualize.py
Connections 21
- gmp entry → Dispatcher: starts · Entry module hands control to the dispatcher.
- Dispatcher → Eval CLI: routes · Dispatches the eval subcommand.
- Dispatcher → Submit CLI: routes · Dispatches the submit subcommand.
- Dispatcher → Status CLI: routes · Dispatches the status subcommand.
- Dispatcher → Plot CLI: routes · Dispatches the plot subcommand.
- Dispatcher → Cleanup: routes · Dispatches the cleanup subcommand.
- Dispatcher → Online CLI: routes · Dispatches the online workflow subcommand.
- Dispatcher → Leaderboard CLI: routes · Dispatches the leaderboard subcommand.
- Dispatcher → Viz CLI: routes · Dispatches the visualization subcommand.
- Eval CLI → Eval client: invokes · CLI delegates execution to the eval runner backend.
- Submit CLI → Submit logic: invokes · CLI delegates submission request construction and sending.
- Online CLI → Online client: invokes · CLI delegates remote task creation and endpoint polling.
- Leaderboard CLI → Leaderboard: invokes · CLI delegates leaderboard operations.
- Viz CLI → Visualizer: invokes · CLI launches the local visualization service.
- Plot CLI → Plot utils: uses · Plot command uses artifact parsing and rendering helpers.
- Eval client → Submit logic: calls server · Evaluation flow relies on backend job/server interactions.
- Eval client → Online client: optional endpoint · Can connect through an online-provided endpoint.
- Visualizer → Plot utils: reads artifacts · Visualizer consumes saved episode outputs and caches derived recordings.
- Visualizer → Cleanup: shares filesystem · Both touch generated local artifacts and cache directories.
- Leaderboard → Submit logic: uses ids · Leaderboard workflow bridges run artifacts and benchmark submission identifiers.
- Online client → Eval client: feeds endpoint · Online service assigns an endpoint consumed by eval.