# openclaw/lobster architecture

> Architecture diagram of the GitHub repository openclaw/lobster, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openclaw/lobster
- Repository: https://github.com/openclaw/lobster

## Overview

1. **What this repo is**
   Lobster is a local-first workflow shell in TypeScript/Node that executes typed pipelines, shell commands, and approval gates. It is designed to be invoked directly by an AI agent or by a human, while keeping auth outside the repo boundary.

2. **Entry points and packaging boundary**
   The runnable entrypoints are the small shims in `bin/` (`lobster.js`, `openclaw.invoke.js`, `clawd.invoke.js`). They forward into the compiled app in `dist/` when available, so the runtime boundary is “CLI process in / node package out,” not a long-running service.

3. **CLI front end**
   `src/cli.ts` is the main command-line surface. It accepts one-liner pipeline expressions, workflow commands, and utility commands like `doctor`. The CLI is the orchestration layer that decides whether input is a shell-style command, a Lobster pipeline, or a workflow invocation.

4. **Command execution model**
   `src/commands/registry.ts` and `src/commands/commands_list.ts` define the command catalog. Commands are split into:
   - **stdlib pipeline stages** in `src/commands/stdlib/` such as `exec`, `where`, `pick`, `map`, `json`, `table`, `sort`, `group_by`, `head`, `dedupe`, `diff_last`, `template`, and `state`
   - **workflow commands** in `src/commands/workflows/`
   This is the main typed-data boundary: commands operate on objects/arrays and can still bridge to shell processes when needed.

5. **Data flow through pipelines**
   Pipeline input can come from the previous step’s `stdout`, `json`, or `jsonl`, and the repo’s README emphasizes “JSON-first” flow instead of text piping. `src/parser.ts` and `src/read_line.ts` handle command parsing and interactive line input, while renderers in `src/renderers/json.ts` turn structured results back into terminal-friendly output.

6. **Process/runtime boundary**
   `src/core/tool_runtime.ts`, `src/runtime.ts`, and `src/shell.ts` separate Lobster’s internal execution model from the host OS shell. This is where shell commands are spawned, stdin/stdout are wired, and command envelopes are normalized. Tests like `core_tool_runtime.test.ts`, `shell.test.ts`, and `tool_envelope_version.test.ts` show this boundary is treated as a compatibility contract.

7. **Workflow engine**
   Workflows live under `src/workflows/` and are also exposed as commands via `src/commands/workflows/`. The file-based workflow runner (`src/workflows/file.ts`) supports step definitions with `run`, `command`, `pipeline`, `stdin`, `env`, `when`, and `approval`. This makes Lobster both a shell and a small workflow engine with explicit step dependencies.

8. **State, approvals, and resumability**
   Persistent state is isolated in `src/state/store.ts`, with workflow resume logic in `src/resume.ts` and approval primitives in `src/sdk/primitives/approve.ts` plus stdlib `approve`. The architecture is intentionally resumable: approval gates can pause execution, and later resumes rehydrate state instead of replaying everything from scratch.

9. **SDK and external embedding boundary**
   `src/sdk/Lobster.ts`, `src/sdk/runtime.ts`, and `src/sdk/primitives/` expose a programmatic interface for embedding Lobster behavior inside another host, especially OpenClaw. This is the boundary that lets external tools call Lobster as a tool/runtime rather than only as a CLI.

10. **OpenClaw integration path**
   The README and `src/commands/stdlib/openclaw_invoke.ts` indicate a dedicated bridge to OpenClaw. The `openclaw.invoke` and `clawd.invoke` shims are meant to make Lobster available as an OpenClaw-native tool without Lobster owning any OAuth or token lifecycle. Auth stays in the environment, not in the workflow engine.

11. **LLM and provider abstraction**
   `src/commands/stdlib/llm_invoke.ts` and `llm_task_invoke.ts` provide model-backed pipeline steps. Provider selection is explicit and environment-driven (`OPENCLAW_URL`, `OPENCLAW_TOKEN`, `LOBSTER_LLM_ADAPTER_URL`, `LOBSTER_PI_LLM_ADAPTER_URL`), so the architecture treats LLMs as pluggable backends rather than a hardcoded service.

12. **Domain-specific workflows and recipes**
   `src/workflows/github_pr_monitor.ts` and `src/recipes/github/pr-monitor.ts` show a higher-level domain layer for GitHub PR monitoring. The repo uses these as examples of reusable, typed workflows that fetch a snapshot, compare against stored state, and emit a change summary.

13. **Input/output integration points**
   There are built-in integrations for Gmail/search/send (`gog_gmail_search`, `gog_gmail_send`) and email triage, showing Lobster is meant to sit between external systems and AI agents. These commands are small adapters; the workflow layer composes them rather than embedding business logic in the CLI.

14. **Verification and compatibility discipline**
   The `test/` suite is broad and architecture-relevant: it covers parsing, command registration, workflow args/env propagation, approval preview/resume, OpenClaw invoke aliases, and workflow file execution. That signals the repo’s core contracts are CLI syntax, structured data flow, and resumable execution semantics.

Main entry points: [`src/commands/stdlib`](https://github.com/openclaw/lobster/tree/main/src/commands/stdlib) (Stdlib stages), [`src/workflows/file.ts`](https://github.com/openclaw/lobster/blob/main/src/workflows/file.ts) (File runner), [`src/cli.ts`](https://github.com/openclaw/lobster/blob/main/src/cli.ts) (CLI), [`src/commands/registry.ts`](https://github.com/openclaw/lobster/blob/main/src/commands/registry.ts) (Registry)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_entry["Entry & packaging"]
  node_node_bin_shims["Bin shims<br/>entrypoint shims"]
  node_node_cli["CLI<br/>command surface<br/>[cli.ts]"]
end

subgraph group_group_pipeline["Pipeline engine"]
  node_node_parser["Parser<br/>input parsing<br/>[parser.ts]"]
  node_node_read_line["Read line<br/>interactive input<br/>[read_line.ts]"]
  node_node_commands_registry["Registry<br/>command catalog<br/>[registry.ts]"]
  node_node_commands_list["Commands list<br/>command manifest<br/>[commands_list.ts]"]
  node_node_stdlib["Stdlib stages<br/>pipeline commands"]
  node_node_shell["Shell bridge<br/>process bridge<br/>[shell.ts]"]
  node_node_tool_runtime["Tool runtime<br/>execution envelope<br/>[tool_runtime.ts]"]
  node_node_renderer_json["JSON renderer<br/>output rendering<br/>[json.ts]"]
end

subgraph group_group_workflow["Workflows & recipes"]
  node_node_workflow_cmds["Workflow cmds<br/>workflow commands"]
  node_node_workflow_registry["Workflow registry<br/>workflow catalog<br/>[registry.ts]"]
  node_node_workflow_file["File runner<br/>workflow engine<br/>[file.ts]"]
  node_node_github_pr_workflow["PR monitor<br/>domain workflow"]
  node_node_recipes["Recipes"]
end

subgraph group_group_runtime["Runtime & state"]
  node_node_runtime["Runtime<br/>[runtime.ts]"]
  node_node_resume["Resume<br/>recovery<br/>[resume.ts]"]
  node_node_state_store[("State store<br/>persistent state<br/>[store.ts]")]
end

subgraph group_group_sdk["SDK & integrations"]
  node_node_sdk_lobster["SDK API<br/>embedding API<br/>[Lobster.ts]"]
  node_node_sdk_runtime["SDK runtime<br/>embedding runtime<br/>[runtime.ts]"]
  node_node_sdk_primitives["Primitives"]
  node_node_llm["LLM adapters<br/>model backends<br/>[llm_invoke.ts]"]
  node_node_openclaw_bridge["OpenClaw bridge<br/>integration bridge<br/>[openclaw_invoke.ts]"]
  node_node_gmail["Gmail adapters<br/>external adapters"]
end

node_node_bin_shims -->|"launches"| node_node_cli
node_node_cli -->|"parses"| node_node_parser
node_node_cli -->|"reads"| node_node_read_line
node_node_cli -->|"dispatches via"| node_node_commands_registry
node_node_commands_registry -->|"loads"| node_node_commands_list
node_node_commands_registry -->|"includes"| node_node_stdlib
node_node_commands_registry -->|"includes"| node_node_workflow_cmds
node_node_stdlib -->|"bridges to"| node_node_shell
node_node_stdlib -->|"renders"| node_node_renderer_json
node_node_stdlib -->|"invokes"| node_node_llm
node_node_stdlib -->|"bridges"| node_node_openclaw_bridge
node_node_stdlib -->|"adapts"| node_node_gmail
node_node_shell -->|"normalized by"| node_node_tool_runtime
node_node_tool_runtime -->|"runs under"| node_node_runtime
node_node_workflow_cmds -->|"exposes"| node_node_workflow_registry
node_node_workflow_registry -->|"executes with"| node_node_workflow_file
node_node_workflow_file -->|"orchestrated by"| node_node_runtime
node_node_workflow_file -->|"supports"| node_node_resume
node_node_resume -->|"persists in"| node_node_state_store
node_node_workflow_file -->|"uses"| node_node_sdk_primitives
node_node_sdk_lobster -->|"wraps"| node_node_sdk_runtime
node_node_sdk_runtime -->|"composes"| node_node_sdk_primitives
node_node_sdk_primitives -->|"stores state"| node_node_state_store
node_node_github_pr_workflow -->|"specializes"| node_node_workflow_file
node_node_recipes -->|"contains"| node_node_github_pr_workflow
node_node_llm -->|"configured by"| node_node_sdk_runtime
node_node_openclaw_bridge -->|"invoked through"| node_node_sdk_lobster
node_node_gmail -->|"composed in"| node_node_workflow_file

click node_node_bin_shims "https://github.com/openclaw/lobster/tree/main/bin"
click node_node_cli "https://github.com/openclaw/lobster/blob/main/src/cli.ts"
click node_node_parser "https://github.com/openclaw/lobster/blob/main/src/parser.ts"
click node_node_read_line "https://github.com/openclaw/lobster/blob/main/src/read_line.ts"
click node_node_commands_registry "https://github.com/openclaw/lobster/blob/main/src/commands/registry.ts"
click node_node_commands_list "https://github.com/openclaw/lobster/blob/main/src/commands/commands_list.ts"
click node_node_stdlib "https://github.com/openclaw/lobster/tree/main/src/commands/stdlib"
click node_node_shell "https://github.com/openclaw/lobster/blob/main/src/shell.ts"
click node_node_tool_runtime "https://github.com/openclaw/lobster/blob/main/src/core/tool_runtime.ts"
click node_node_renderer_json "https://github.com/openclaw/lobster/blob/main/src/renderers/json.ts"
click node_node_workflow_cmds "https://github.com/openclaw/lobster/tree/main/src/commands/workflows"
click node_node_workflow_registry "https://github.com/openclaw/lobster/blob/main/src/workflows/registry.ts"
click node_node_workflow_file "https://github.com/openclaw/lobster/blob/main/src/workflows/file.ts"
click node_node_github_pr_workflow "https://github.com/openclaw/lobster/blob/main/src/workflows/github_pr_monitor.ts"
click node_node_recipes "https://github.com/openclaw/lobster/tree/main/src/recipes"
click node_node_runtime "https://github.com/openclaw/lobster/blob/main/src/runtime.ts"
click node_node_resume "https://github.com/openclaw/lobster/blob/main/src/resume.ts"
click node_node_state_store "https://github.com/openclaw/lobster/blob/main/src/state/store.ts"
click node_node_sdk_lobster "https://github.com/openclaw/lobster/blob/main/src/sdk/Lobster.ts"
click node_node_sdk_runtime "https://github.com/openclaw/lobster/blob/main/src/sdk/runtime.ts"
click node_node_sdk_primitives "https://github.com/openclaw/lobster/tree/main/src/sdk/primitives"
click node_node_llm "https://github.com/openclaw/lobster/blob/main/src/commands/stdlib/llm_invoke.ts"
click node_node_openclaw_bridge "https://github.com/openclaw/lobster/blob/main/src/commands/stdlib/openclaw_invoke.ts"
click node_node_gmail "https://github.com/openclaw/lobster/blob/main/src/commands/stdlib/gog_gmail_search.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_bin_shims,node_node_cli toneBlue
class node_node_parser,node_node_read_line,node_node_commands_registry,node_node_commands_list,node_node_stdlib,node_node_shell,node_node_tool_runtime,node_node_renderer_json toneAmber
class node_node_workflow_cmds,node_node_workflow_registry,node_node_workflow_file,node_node_github_pr_workflow,node_node_recipes toneMint
class node_node_runtime,node_node_resume,node_node_state_store toneRose
class node_node_sdk_lobster,node_node_sdk_runtime,node_node_sdk_primitives,node_node_llm,node_node_openclaw_bridge,node_node_gmail toneIndigo
```

## Components

### Entry & packaging

Runnable shims and CLI surface.

- [Bin shims](https://github.com/openclaw/lobster/tree/main/bin): entrypoint shims · Small Node launchers that hand off to the compiled app. · `bin`
- [CLI](https://github.com/openclaw/lobster/blob/main/src/cli.ts): command surface · Main command-line orchestration layer for pipelines, workflows, and utilities. · `src/cli.ts`

### Pipeline engine

Parsing, command catalog, structured rendering, and shell bridging.

- [Parser](https://github.com/openclaw/lobster/blob/main/src/parser.ts): input parsing · Parses one-liners and command expressions from the CLI and interactive input. · `src/parser.ts`
- [Read line](https://github.com/openclaw/lobster/blob/main/src/read_line.ts): interactive input · Interactive line reader used by the CLI shell experience. · `src/read_line.ts`
- [Registry](https://github.com/openclaw/lobster/blob/main/src/commands/registry.ts): command catalog · Registers commands and exposes the command catalog. · `src/commands/registry.ts`
- [Commands list](https://github.com/openclaw/lobster/blob/main/src/commands/commands_list.ts): command manifest · Defines the available command set and categories. · `src/commands/commands_list.ts`
- [Stdlib stages](https://github.com/openclaw/lobster/tree/main/src/commands/stdlib): pipeline commands · Typed data pipeline stages such as exec, json, table, map, where, and similar transforms. · `src/commands/stdlib`
- [Shell bridge](https://github.com/openclaw/lobster/blob/main/src/shell.ts): process bridge · Spawns host shell processes and wires stdin/stdout boundaries. · `src/shell.ts`
- [Tool runtime](https://github.com/openclaw/lobster/blob/main/src/core/tool_runtime.ts): execution envelope · Normalizes command envelopes and coordinates tool execution contracts. · `src/core/tool_runtime.ts`
- [JSON renderer](https://github.com/openclaw/lobster/blob/main/src/renderers/json.ts): output rendering · Renders structured pipeline results back to terminal-friendly JSON output. · `src/renderers/json.ts`

### Workflows & recipes

File-based workflows, workflow commands, and domain recipes.

- [Workflow cmds](https://github.com/openclaw/lobster/tree/main/src/commands/workflows): workflow commands · Command wrappers that expose workflows through the CLI catalog. · `src/commands/workflows`
- [Workflow registry](https://github.com/openclaw/lobster/blob/main/src/workflows/registry.ts): workflow catalog · Registers available workflows and workflow entry points. · `src/workflows/registry.ts`
- [File runner](https://github.com/openclaw/lobster/blob/main/src/workflows/file.ts): workflow engine · Runs file-defined workflows with steps, conditions, approvals, and stdin/env wiring. · `src/workflows/file.ts`
- [PR monitor](https://github.com/openclaw/lobster/blob/main/src/workflows/github_pr_monitor.ts): domain workflow · GitHub pull-request monitoring workflow and summary pipeline. · `src/workflows/github_pr_monitor.ts`
- [Recipes](https://github.com/openclaw/lobster/tree/main/src/recipes): workflow recipes · Higher-level reusable workflow recipes built on the workflow engine. · `src/recipes`

### Runtime & state

Execution runtime, resumability, approvals, and persistent state.

- [Runtime](https://github.com/openclaw/lobster/blob/main/src/runtime.ts): execution runtime · Coordinates local-first execution and host-process integration. · `src/runtime.ts`
- [Resume](https://github.com/openclaw/lobster/blob/main/src/resume.ts): recovery · Rehydrates paused workflows and continues resumable executions. · `src/resume.ts`
- [State store](https://github.com/openclaw/lobster/blob/main/src/state/store.ts): persistent state · Persistent storage for workflow state and resumability. · `src/state/store.ts`

### SDK & integrations

Embedding APIs, primitives, and external adapters.

- [SDK API](https://github.com/openclaw/lobster/blob/main/src/sdk/Lobster.ts): embedding API · Programmatic API for embedding Lobster in another host. · `src/sdk/Lobster.ts`
- [SDK runtime](https://github.com/openclaw/lobster/blob/main/src/sdk/runtime.ts): embedding runtime · SDK-facing runtime wrapper for external callers. · `src/sdk/runtime.ts`
- [Primitives](https://github.com/openclaw/lobster/tree/main/src/sdk/primitives): sdk primitives · Reusable approve, exec, diff, and state primitives for embedding and workflows. · `src/sdk/primitives`
- [LLM adapters](https://github.com/openclaw/lobster/blob/main/src/commands/stdlib/llm_invoke.ts): model backends · Model-invocation commands and provider abstractions for pluggable LLM backends. · `src/commands/stdlib/llm_invoke.ts`
- [OpenClaw bridge](https://github.com/openclaw/lobster/blob/main/src/commands/stdlib/openclaw_invoke.ts): integration bridge · Dedicated invocation bridge for OpenClaw-native usage and token/env-based auth. · `src/commands/stdlib/openclaw_invoke.ts`
- [Gmail adapters](https://github.com/openclaw/lobster/blob/main/src/commands/stdlib/gog_gmail_search.ts): external adapters · Email search and send adapters used by workflow composition. · `src/commands/stdlib/gog_gmail_search.ts`

## Connections

- Bin shims → CLI: launches · Entry shims invoke the CLI runtime.
- CLI → Parser: parses · CLI turns input into command or pipeline expressions.
- CLI → Read line: reads · CLI supports interactive line input.
- CLI → Registry: dispatches via · CLI resolves commands through the registry.
- Registry → Commands list: loads · Registry is populated from the command manifest.
- Registry → Stdlib stages: includes · Registry exposes stdlib pipeline stages.
- Registry → Workflow cmds: includes · Registry exposes workflow commands.
- Stdlib stages → Shell bridge: bridges to · Shell execution is used by command stages that cross the OS boundary.
- Stdlib stages → JSON renderer: renders · Structured outputs are rendered for terminal consumption.
- Stdlib stages → LLM adapters: invokes · Some stages delegate to model backends.
- Stdlib stages → OpenClaw bridge: bridges · Stdlib includes OpenClaw invocation support.
- Stdlib stages → Gmail adapters: adapts · Stdlib includes external email adapters.
- Shell bridge → Tool runtime: normalized by · Process execution is mediated through tool runtime envelopes.
- Tool runtime → Runtime: runs under · Tool runtime sits inside the broader runtime layer.
- Workflow cmds → Workflow registry: exposes · Workflow commands are backed by the workflow registry.
- Workflow registry → File runner: executes with · Registered workflows use the file-based workflow runner.
- File runner → Runtime: orchestrated by · Workflow execution is coordinated by the runtime.
- File runner → Resume: supports · Workflow runs can be paused and resumed.
- Resume → State store: persists in · Resume logic rehydrates from persistent state.
- File runner → Primitives: uses · Workflows rely on shared primitives for approve/exec/state behavior.
- SDK API → SDK runtime: wraps · The public SDK API is implemented on top of the SDK runtime.
- SDK runtime → Primitives: composes · SDK runtime composes shared primitives.
- Primitives → State store: stores state · Primitives interact with persistent workflow state.
- PR monitor → File runner: specializes · GitHub PR workflow is a concrete workflow built on the runner.
- Recipes → PR monitor: contains · Recipes include the PR monitor domain flow.
- LLM adapters → SDK runtime: configured by · Provider selection is driven through runtime/environment configuration.
- OpenClaw bridge → SDK API: invoked through · OpenClaw bridge exposes Lobster as a tool/runtime.
- Gmail adapters → File runner: composed in · Email adapters are orchestrated by workflows rather than CLI logic.
