How openclaw/lobster works
1. What this repo is
Lobster is a local-first workflow shell in TypeScript/Node that executes typed pipelines, shell commands, and approval gates. It is designed to be invoked directly by an AI agent or by a human, while keeping auth outside the repo boundary.
2. Entry points and packaging boundary
The runnable entrypoints are the small shims in bin/ (lobster.js, openclaw.invoke.js, clawd.invoke.js). They forward into the compiled app in dist/ when available, so the runtime boundary is “CLI process in / node package out,” not a long-running service.
3. CLI front end
src/cli.ts is the main command-line surface. It accepts one-liner pipeline expressions, workflow commands, and utility commands like doctor. The CLI is the orchestration layer that decides whether input is a shell-style command, a Lobster pipeline, or a workflow invocation.
4. Command execution model
src/commands/registry.ts and src/commands/commands_list.ts define the command catalog. Commands are split into:
- stdlib pipeline stages in src/commands/stdlib/ such as exec, where, pick, map, json, table, sort, group_by, head, dedupe, diff_last, template, and state
- workflow commands in src/commands/workflows/
This is the main typed-data boundary: commands operate on objects/arrays and can still bridge to shell processes when needed.
5. Data flow through pipelines
Pipeline input can come from the previous step’s stdout, json, or jsonl, and the repo’s README emphasizes “JSON-first” flow instead of text piping. src/parser.ts and src/read_line.ts handle command parsing and interactive line input, while renderers in src/renderers/json.ts turn structured results back into terminal-friendly output.
6. Process/runtime boundary
src/core/tool_runtime.ts, src/runtime.ts, and src/shell.ts separate Lobster’s internal execution model from the host OS shell. This is where shell commands are spawned, stdin/stdout are wired, and command envelopes are normalized. Tests like core_tool_runtime.test.ts, shell.test.ts, and tool_envelope_version.test.ts show this boundary is treated as a compatibility contract.
7. Workflow engine
Workflows live under src/workflows/ and are also exposed as commands via src/commands/workflows/. The file-based workflow runner (src/workflows/file.ts) supports step definitions with run, command, pipeline, stdin, env, when, and approval. This makes Lobster both a shell and a small workflow engine with explicit step dependencies.
8. State, approvals, and resumability
Persistent state is isolated in src/state/store.ts, with workflow resume logic in src/resume.ts and approval primitives in src/sdk/primitives/approve.ts plus stdlib approve. The architecture is intentionally resumable: approval gates can pause execution, and later resumes rehydrate state instead of replaying everything from scratch.
9. SDK and external embedding boundary
src/sdk/Lobster.ts, src/sdk/runtime.ts, and src/sdk/primitives/ expose a programmatic interface for embedding Lobster behavior inside another host, especially OpenClaw. This is the boundary that lets external tools call Lobster as a tool/runtime rather than only as a CLI.
10. OpenClaw integration path
The README and src/commands/stdlib/openclaw_invoke.ts indicate a dedicated bridge to OpenClaw. The openclaw.invoke and clawd.invoke shims are meant to make Lobster available as an OpenClaw-native tool without Lobster owning any OAuth or token lifecycle. Auth stays in the environment, not in the workflow engine.
11. LLM and provider abstraction
src/commands/stdlib/llm_invoke.ts and llm_task_invoke.ts provide model-backed pipeline steps. Provider selection is explicit and environment-driven (OPENCLAW_URL, OPENCLAW_TOKEN, LOBSTER_LLM_ADAPTER_URL, LOBSTER_PI_LLM_ADAPTER_URL), so the architecture treats LLMs as pluggable backends rather than a hardcoded service.
12. Domain-specific workflows and recipes
src/workflows/github_pr_monitor.ts and src/recipes/github/pr-monitor.ts show a higher-level domain layer for GitHub PR monitoring. The repo uses these as examples of reusable, typed workflows that fetch a snapshot, compare against stored state, and emit a change summary.
13. Input/output integration points
There are built-in integrations for Gmail/search/send (gog_gmail_search, gog_gmail_send) and email triage, showing Lobster is meant to sit between external systems and AI agents. These commands are small adapters; the workflow layer composes them rather than embedding business logic in the CLI.
14. Verification and compatibility discipline
The test/ suite is broad and architecture-relevant: it covers parsing, command registration, workflow args/env propagation, approval preview/resume, OpenClaw invoke aliases, and workflow file execution. That signals the repo’s core contracts are CLI syntax, structured data flow, and resumable execution semantics.
Main entry points: stdlib Stdlib stages · file.ts File runner · cli.ts CLI · registry.ts Registry
Components 24
Entry & packaging
Runnable shims and CLI surface.
Pipeline engine
Parsing, command catalog, structured rendering, and shell bridging.
- Parser · Parses one-liners and command expressions from the CLI and interactive input.
src/parser.ts - Read line · Interactive line reader used by the CLI shell experience.
src/read_line.ts - Registry · Registers commands and exposes the command catalog.
src/commands/registry.ts - Commands list · Defines the available command set and categories.
src/commands/commands_list.ts - Stdlib stages · Typed data pipeline stages such as exec, json, table, map, where, and similar transforms.
src/commands/stdlib - Shell bridge · Spawns host shell processes and wires stdin/stdout boundaries.
src/shell.ts - Tool runtime · Normalizes command envelopes and coordinates tool execution contracts.
src/core/tool_runtime.ts - JSON renderer · Renders structured pipeline results back to terminal-friendly JSON output.
src/renderers/json.ts
Workflows & recipes
File-based workflows, workflow commands, and domain recipes.
- Workflow cmds · Command wrappers that expose workflows through the CLI catalog.
src/commands/workflows - Workflow registry · Registers available workflows and workflow entry points.
src/workflows/registry.ts - File runner · Runs file-defined workflows with steps, conditions, approvals, and stdin/env wiring.
src/workflows/file.ts - PR monitor · GitHub pull-request monitoring workflow and summary pipeline.
src/workflows/github_pr_monitor.ts - Recipes · Higher-level reusable workflow recipes built on the workflow engine.
src/recipes
Runtime & state
Execution runtime, resumability, approvals, and persistent state.
- Runtime · Coordinates local-first execution and host-process integration.
src/runtime.ts - Resume · Rehydrates paused workflows and continues resumable executions.
src/resume.ts - State store · Persistent storage for workflow state and resumability.
src/state/store.ts
SDK & integrations
Embedding APIs, primitives, and external adapters.
- SDK API · Programmatic API for embedding Lobster in another host.
src/sdk/Lobster.ts - SDK runtime · SDK-facing runtime wrapper for external callers.
src/sdk/runtime.ts - Primitives · Reusable approve, exec, diff, and state primitives for embedding and workflows.
src/sdk/primitives - LLM adapters · Model-invocation commands and provider abstractions for pluggable LLM backends.
src/commands/stdlib/llm_invoke.ts - OpenClaw bridge · Dedicated invocation bridge for OpenClaw-native usage and token/env-based auth.
src/commands/stdlib/openclaw_invoke.ts - Gmail adapters · Email search and send adapters used by workflow composition.
src/commands/stdlib/gog_gmail_search.ts
Connections 28
- Bin shims → CLI: launches · Entry shims invoke the CLI runtime.
- CLI → Parser: parses · CLI turns input into command or pipeline expressions.
- CLI → Read line: reads · CLI supports interactive line input.
- CLI → Registry: dispatches via · CLI resolves commands through the registry.
- Registry → Commands list: loads · Registry is populated from the command manifest.
- Registry → Stdlib stages: includes · Registry exposes stdlib pipeline stages.
- Registry → Workflow cmds: includes · Registry exposes workflow commands.
- Stdlib stages → Shell bridge: bridges to · Shell execution is used by command stages that cross the OS boundary.
- Stdlib stages → JSON renderer: renders · Structured outputs are rendered for terminal consumption.
- Stdlib stages → LLM adapters: invokes · Some stages delegate to model backends.
- Stdlib stages → OpenClaw bridge: bridges · Stdlib includes OpenClaw invocation support.
- Stdlib stages → Gmail adapters: adapts · Stdlib includes external email adapters.
- Shell bridge → Tool runtime: normalized by · Process execution is mediated through tool runtime envelopes.
- Tool runtime → Runtime: runs under · Tool runtime sits inside the broader runtime layer.
- Workflow cmds → Workflow registry: exposes · Workflow commands are backed by the workflow registry.
- Workflow registry → File runner: executes with · Registered workflows use the file-based workflow runner.
- File runner → Runtime: orchestrated by · Workflow execution is coordinated by the runtime.
- File runner → Resume: supports · Workflow runs can be paused and resumed.
- Resume → State store: persists in · Resume logic rehydrates from persistent state.
- File runner → Primitives: uses · Workflows rely on shared primitives for approve/exec/state behavior.
- SDK API → SDK runtime: wraps · The public SDK API is implemented on top of the SDK runtime.
- SDK runtime → Primitives: composes · SDK runtime composes shared primitives.
- Primitives → State store: stores state · Primitives interact with persistent workflow state.
- PR monitor → File runner: specializes · GitHub PR workflow is a concrete workflow built on the runner.
- Recipes → PR monitor: contains · Recipes include the PR monitor domain flow.
- LLM adapters → SDK runtime: configured by · Provider selection is driven through runtime/environment configuration.
- OpenClaw bridge → SDK API: invoked through · OpenClaw bridge exposes Lobster as a tool/runtime.
- Gmail adapters → File runner: composed in · Email adapters are orchestrated by workflows rather than CLI logic.