How katanemo/plano works
Rust dataplane core
The primary production system lives under crates/. The main executable is crates/brightstaff, which appears to be the server/runtime that accepts traffic, applies routing/filtering/state, and emits observability data. Supporting crates split reusable protocol logic and policy logic:
• crates/common for shared config, routing, HTTP, tokenization, PII, rate limiting, tracing, and provider abstractions.
• crates/hermesllm for provider adapters and request/response transforms.
• crates/llm_gateway for model-routing gateway behavior.
• crates/prompt_gateway for prompt/filter/tool orchestration behavior.
Request flow through the proxy
The runtime is organized around HTTP request handling and streaming responses. Incoming traffic is routed through router/, then into handlers/ for specific features such as agents, LLM handling, function calling, model selection, and debug endpoints. This suggests a single dataplane that can act as:
• an agent orchestrator,
• a model gateway,
• or a filter/guardrail pipeline,
depending on configuration.
Agent orchestration subsystem
The handlers/agents and router/orchestrator* code indicates first-class multi-agent routing. Agent requests are selected and dispatched based on descriptions or routing logic, with orchestrator/pipeline/selector components separating:
• intent or agent selection,
• orchestration of one or more downstream agents,
• and JSON-RPC-style or OpenAI-compatible request/response handling.
This is the boundary where Plano coordinates multiple agent services without app code needing explicit routing logic.
LLM routing gateway
crates/llm_gateway and the corresponding handlers under brightstaff/src/handlers/llm indicate a dedicated model-routing path. The repo supports routing by explicit model name, aliases, and preference-based selection. This is distinct from agent orchestration: here the proxy decides *which model/provider to use* for a request, while preserving a unified API surface for clients.
Prompt gateway and filter-chain behavior
crates/prompt_gateway plus the demos/filter_chains material show a separate concern: applying guardrails, context building, memory hooks, tool invocation, and request/response filtering around prompts. The filter chain is not just a simple middleware; it is a structured pipeline with HTTP context, stream context, filter context, and tool/context helpers. This is the repository’s safety/moderation/memory boundary.
Signals, tracing, and observability
Observability is a first-class subsystem in Rust under brightstaff/src/signals, metrics, and tracing, plus shared trace utilities in common/src/traces. The codebase treats “signals” as domain-specific analytical events beyond raw logs/traces, with analyzers for interaction, execution, and environment conditions. There is also PostHog and OpenTelemetry export plumbing, so requests can be observed end-to-end without instrumenting each agent manually.
State and session persistence
Session or conversation state is abstracted in brightstaff/src/state and session_cache, with in-memory and Redis/PostgreSQL backends. This is important because several features in the repo rely on preserving conversational or response state across turns, especially for orchestrated or responses-style APIs. State management is a separate data boundary from routing and from provider communication.
Python CLI and operational tooling
The cli/planoai package is the operator-facing control plane. It handles init/config generation, template sync, native binary launching, Docker-based workflows, tracing helpers, and observability/reporting commands. It also bundles template YAMLs for common patterns. In architecture terms, the CLI prepares and launches the runtime rather than serving end-user traffic itself.
Configuration, templates, and deployment surfaces
Runtime behavior is driven by YAML configs in config/ and many demo-specific configs. There are clear deployment artifacts for Docker Compose, Envoy templates, Prometheus/Grafana dashboards, and supervised local environments. This indicates the dataplane is meant to be deployed as an infrastructure component, not embedded in application code. The config/envoy.template.yaml and related files suggest Envoy is part of the external proxy/deployment model.
Docs site and product site
There are two Next.js apps under apps/: apps/www for the main documentation/product site and apps/katanemo-www for the marketing site. The docs app also includes a Sanity studio route, blog content, SEO endpoints, and a lot of explainer UI around routing, observability, and concepts. These are not part of the runtime dataplane, but they document and manage the product’s architecture and examples.
Examples, demos, and verification
demos/ is large and intentionally mirrors the major architectural modes: agent orchestration, filter chains, LLM routing, provider integrations, and getting started examples. tests/ includes integration, end-to-end, REST/Hurl, and parity tests that validate the proxy behavior against model routing and prompt gateway scenarios. This test structure reinforces the repo’s boundaries by exercising each subsystem as an externally visible service.
Overall architectural picture
The repo’s central idea is a configurable AI middleware layer that sits between client applications and downstream LLMs/agents. The Rust runtime handles request interception, routing, filtering, state, and observability; hermesllm adapts provider APIs; the Python CLI and YAML configs define deployment/runtime behavior; and the web/docs/demo assets demonstrate how to compose these pieces into agentic applications.
Main entry points: main.rs Brightstaff · handlers Handlers · main.py Plano CLI · src Common core
Components 21
Control plane
Operator tooling, configuration, and deployment surfaces.
- Plano CLI · Python command-line tool for init, config generation, launch, tracing, and observability workflows.
cli/planoai/main.py - CLI templates · Opinionated runtime templates used by the operator to generate ready-made Plano configs.
cli/planoai/templates - Config bundle · YAML schema, environment, and local deployment templates that shape runtime behavior.
config
Rust runtime
Dataplane server and shared protocol logic.
- Brightstaff · Primary Rust dataplane executable that receives requests and dispatches them through routing, handlers, state, and observability.
crates/brightstaff/src/main.rs - Router core · HTTP and orchestrator routing layer that selects request paths and model/orchestrator flows.
crates/brightstaff/src/router - Handlers · Feature handlers for agents, LLMs, model selection, function calling, debug, and response flows.
crates/brightstaff/src/handlers - Agent orchestration · Agent selection and orchestration pipeline, including selector, pipeline, and JSON-RPC handling.
crates/brightstaff/src/handlers/agents - LLM handler · LLM request handling and model selection path for unified client-facing model APIs.
crates/brightstaff/src/handlers/llm - State store · Conversation and response state persistence with memory and Redis/PostgreSQL backends.
crates/brightstaff/src/state - Session cache · In-process and Redis-backed cache for conversational/session continuity.
crates/brightstaff/src/session_cache - Signals · Domain-specific analyzers and event schemas for interaction, execution, and environment signals.
crates/brightstaff/src/signals - Tracing · Trace initialization and exporters that connect runtime activity to observability backends.
crates/brightstaff/src/tracing - Common core · Shared config, HTTP, routing, rate limiting, tokenization, PII, traces, and provider abstractions.
crates/common/src - HermesLLM · Provider normalization layer for OpenAI, Anthropic, Bedrock, responses APIs, and streaming transforms.
crates/hermesllm/src - LLM gateway · Model-routing gateway logic and filter/stream contexts used by the runtime and demos.
crates/llm_gateway/src - Prompt gateway · Structured prompt/filter pipeline with HTTP context, stream context, tools, and filter orchestration.
crates/prompt_gateway/src
Docs and demos
Documentation site, marketing site, and executable examples.
- Docs site · Primary documentation and product site with concepts, guides, studio, and API content.
apps/www/src - Marketing site · Secondary web app for the public-facing product/brand site.
apps/katanemo-www/src - Demos · Executable examples covering agent orchestration, filter chains, LLM routing, and integrations.
demos - Skills · Operational and pattern-oriented skill docs that encode usage, deployment, and configuration recipes.
skills
Verification
Tests and parity checks that exercise the proxy behaviors.
- Tests · End-to-end, Hurl, REST, and parity tests that validate gateway behavior externally.
tests
Connections 26
- Plano CLI → CLI templates: generates · CLI uses templates to scaffold common Plano configurations.
- Plano CLI → Config bundle: writes · CLI creates and validates runtime configuration and deployment files.
- Plano CLI → Brightstaff: launches · Operator starts the Rust runtime locally or in containers.
- Brightstaff → Router core: routes through · Server entrypoint forwards requests into the router layer.
- Router core → Handlers: dispatches · Router selects the appropriate feature handler for each request.
- Handlers → Agent orchestration: includes · Agent-specific paths are handled by the orchestration subsystem.
- Handlers → LLM handler: includes · Model-centric paths are handled by the LLM gateway handler.
- Handlers → Prompt gateway: includes · Prompt/filter flows are processed through the prompt gateway pipeline.
- Agent orchestration → Common core: uses · Agent flows share routing, HTTP, and provider primitives.
- LLM handler → HermesLLM: adapts via · LLM requests and responses are normalized through provider adapters.
- LLM handler → LLM gateway: routes via · Model selection and alias handling are delegated to the gateway library.
- Prompt gateway → Common core: uses · Filter-chain components reuse shared context and utility code.
- Brightstaff → State store: persists · Runtime stores and reloads conversation or response state.
- Brightstaff → Session cache: caches · Server keeps turn/session data hot for conversational continuity.
- Brightstaff → Signals: emits · Runtime emits domain signals for analysis and telemetry.
- Brightstaff → Tracing: exports · Server initializes tracing and sends spans to observability backends.
- Brightstaff → Common core: shares · Server depends on shared configuration, routing, HTTP, and trace utilities.
- HermesLLM → Common core: shares · Provider adapters reuse shared API and trace helpers.
- Docs site → Demos: documents · Docs site references example architectures and runnable workflows.
- Docs site → Config bundle: explains · Documentation is centered on configuration, deployment, and runtime behavior.
- Demos → Brightstaff: exercises · Demos drive the runtime through its proxy and orchestration modes.
- Tests → Brightstaff: validates · Integration and end-to-end tests exercise the Rust dataplane externally.
- Tests → Prompt gateway: validates · Dedicated tests cover prompt gateway and filter-chain behavior.
- Tests → LLM gateway: validates · Gateway-specific tests verify model routing and selection.
- Skills → Plano CLI: guides · Skill docs encode how to use and operate the CLI.
- Skills → Config bundle: guides · Skill docs explain configuration and deployment patterns.