mizcausevic-dev/agentobserve

Purpose and principal workflow: AgentObserve is an operations API for observing and governing AI-agent fleets, with run traces, cost and latency checks, SLA scoring, and operational posture outputs. The principal workflow begins when an external agent runtime submits a completed run to POST /api/ingest/run; separate evaluation endpoints let API clients request governance checks. The sampled ingest handler validates and summarizes spans but explicitly does not persist the run or invoke the detectors.

Components

HTTP application and API

• Express application — src/index.ts: Builds the Express app, installs middleware, exposes health and Swagger routes, mounts API routers, and listens when run as the main module.

• Run intake and evaluation routes — src/routes/evaluate.ts: Defines run ingestion and cost, regression, SLA, and combined run-audit endpoints; validates request bodies and calls governance functions for evaluation routes.

• Fleet, run, incident, and dashboard reads — src/routes/agents.ts: Serves agent and baseline details, recent and full run records, incidents, and a calculated fleet summary from in-memory data.

• Request schemas — src/schemas/validation-schemas.ts: Defines Zod payload and span validation, including the accepted run-ingestion and evaluation request shapes.

• API documentation — src/docs/swagger.ts: Provides the OpenAPI specification mounted at /docs; its specification contents were not sampled.

Governance evaluations

• Cost and budget evaluation — src/governance/cost-anomaly.ts: Compares submitted run cost with an agent’s baseline and monthly budget, then returns a status, issues, and recommended action.

• Regression evaluation — src/governance/regression-detector.ts: Scores latency, output-length drift, repeated tool calls, retries, and terminal status against baseline data and configured thresholds.

• SLA evaluation — src/governance/sla-evaluator.ts: Assesses agent latency and success-rate targets, computes remaining error budget, and returns compliance status and guidance.

• Combined run audit — src/governance/run-audit.ts: The README describes a combined per-run posture audit using regression and cost checks; implementation details were not sampled.

Domain data and state

• Agent fleet registry — src/data/agents.ts: Defines the in-memory agent catalog, ownership, operating status, budget and SLA metrics, and agent lookup.

• Runs, baselines, and incidents — src/data/runs.ts: Holds in-memory run traces, baseline metrics, and incident records used by read routes and governance logic; ingestion does not persist into this state in the sampled handler.

External participants

• Agent runtime — null repository path: External producer that submits a completed run and its spans.

• API clients and fleet operators — null repository path: External consumers of evaluation and fleet-read APIs; the README frames these outputs as supporting on-call and fleet reviews.

Relationships

• Agent runtime -> Express application: Sends the completed-run HTTP request; the app parses JSON and mounts the intake routes.

• Express application -> Run intake and evaluation routes: Dispatches requests under /api/ingest and /api/evaluate to the corresponding routers.

• Run intake and evaluation routes -> Request schemas: Validates ingestion and evaluation payloads with Zod before handling them.

• Run intake and evaluation routes -> Agent runtime: Returns an ingestion acknowledgment with run and agent IDs, span count, and summed cost and latency; it does not write the submitted run to the in-memory run records.

• API clients and fleet operators -> Express application: Call the mounted fleet-read and governance endpoints over HTTP.

• Express application -> Fleet, run, incident, and dashboard reads: Dispatches /api/agents, /api/runs, /api/incidents, and /api/dashboard requests to the read routers.

• Fleet, run, incident, and dashboard reads -> Agent fleet registry: Reads agent records and resolves agent IDs and names.

• Fleet, run, incident, and dashboard reads -> Runs, baselines, and incidents: Reads runs, baselines, and incidents and derives summary counts and metrics.

• Run intake and evaluation routes -> Cost and budget evaluation: Calls the cost evaluator for /cost-anomaly requests.

• Run intake and evaluation routes -> Regression evaluation: Calls the regression evaluator for /regression requests.

• Run intake and evaluation routes -> SLA evaluation: Calls the SLA evaluator for /sla requests.

• Run intake and evaluation routes -> Combined run audit: Calls the audit function for /run-audit requests; the route supplies a run ID and returns not-found when no audit result is produced.

• Regression evaluation -> Agent fleet registry: Looks up the requested agent.

• Regression evaluation -> Runs, baselines, and incidents: Reads the matching run and baseline to calculate regression findings.

• Cost and budget evaluation -> Agent fleet registry: Reads the agent’s monthly spend and budget.

• Cost and budget evaluation -> Runs, baselines, and incidents: Reads the agent’s baseline cost P95.

• SLA evaluation -> Agent fleet registry: Reads the agent’s observed metrics and SLA thresholds.

• Combined run audit -> Regression evaluation: The README describes regression checks as part of the combined audit; detailed call wiring is not established by the sampled source.

• Combined run audit -> Cost and budget evaluation: The README describes cost checks as part of the combined audit; detailed call wiring is not established by the sampled source.

Coverage limits: The combined audit implementation was not sampled, so its exact detector calls and decision logic cannot be confirmed. The documented API includes additional endpoint behavior not visible in the supplied route excerpts. The repository sample shows in-memory example data and no persistent ingestion path; it does not establish how production telemetry, incident routing, or external alert delivery would work.