How ondc-official/automation-kb-studio works
ai-harness-evaluator probes a declared knowledge manifest against a black-box model or knowledge source, then reports coverage, canary confabulation, and self-consistency without claiming factual correctness. The CLI offers scripted probes; Studio supports folder-first KB authoring, evaluation runs, coverage review, and access-controlled proposal workflows. Core operations call the source and judge through neutral seams, while provider packages adapt external APIs. Studio server excerpt confirms several persistence and policy calls; links for unsampled run orchestration and provider selection are left out.
Main entry points: server.ts Studio API · main.tsx Studio UI · httpClient.ts HTTP client · coverage.ts Coverage probe
Components 27
User interfaces
- CLI
packages/cli/src/cli.ts - Studio UI
packages/studio/src/ui/main.tsx - Coverage view
packages/studio/src/ui/pages/coverage/useCoveragePage.ts - KB authoring
packages/studio/src/ui/pages/author/useAuthorPage.ts - Run evaluation
packages/studio/src/ui/pages/evaluate/useEvaluatePage.ts - Admin review
packages/studio/src/ui/pages/admin/useAdminPage.ts - HTTP client
packages/studio/src/ui/services/httpClient.ts
Evaluation engine
- Core API
packages/core/src/index.ts - Coverage probe
packages/core/src/operations/knowledge/coverage.ts - Knowledge probes
packages/core/src/operations/knowledge - Run events
packages/core/src/runtime/run.ts - Subject profile
packages/core/src/profile/profile.ts - Event reporter
packages/reporter/src/index.ts
Model adapters
- OpenAI adapter
packages/provider-openai/src/openai.ts - Anthropic adapter
packages/provider-anthropic/src/anthropic.ts
Studio services
- Studio API
packages/studio/src/server.ts - Run service
packages/studio/src/runner.ts - Folder manifest
packages/studio/src/manifest-folder.ts - LLM factory
packages/studio/src/llm-factory.ts
Studio state
- Workspace store
packages/studio/src/store.ts - Studio database
packages/studio/src/db.ts - Revision history
packages/studio/src/history.ts - Proposal workflow
packages/studio/src/proposals.ts - Access policy
packages/studio/src/access.ts
Other components
- Operator
- Model backend
- KB folder
Connections 23
- Operator → CLI: runs probes
- CLI → Core API: invokes probes
- Operator → Studio UI: uses Studio
- Studio UI → Coverage view: opens coverage
- Studio UI → KB authoring: edits topics
- Studio UI → Run evaluation: starts runs
- Studio UI → Admin review: reviews access
- Coverage view → HTTP client: requests reports
- KB authoring → HTTP client: sends edits
- Run evaluation → HTTP client: controls runs
- Admin review → HTTP client: sends decisions
- HTTP client → Studio API: calls API
- KB folder → Folder manifest: imports topics
- Studio API → Workspace store: writes topics
- Studio API → Revision history: appends revisions
- Studio API → Proposal workflow: handles proposals
- Studio API → Access policy: checks policy
- Studio API → Studio database: initializes indexes
- Coverage probe → Run events: emits run
- Coverage probe → Knowledge probes: uses knowledge seams
- Run events → Event reporter: streams events
- Model backend → OpenAI adapter: serves requests
- Model backend → Anthropic adapter: serves requests