# microsoft/waza architecture

> Architecture diagram of the GitHub repository microsoft/waza, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/waza
- Repository: https://github.com/microsoft/waza

## Overview

1. **Repository shape and purpose**
   - This is primarily a **Go CLI product** for building and evaluating AI agent skills.
   - The main runtime is `cmd/waza`, supported by a large `internal/` library layer.
   - The repo also contains a **browser dashboard** (`web/`) and a **documentation site** (`site/`), plus reusable **example eval suites** and **skills**.

2. **Top-level execution boundary**
   - The CLI is the entrypoint for almost everything: `init`, `new`, `run`, `grade`, `check`, `suggest`, `compare`, `coverage`, `tokens`, `serve`, `results`, `dev`, and `session`.
   - `cmd/waza` parses flags and dispatches into internal packages; it does not hold business logic itself.
   - `waza` can be used both in a **project workspace** with `skills/` + `evals/`, or in **standalone skill directories**.

3. **Evaluation pipeline is the core subsystem**
   - The central flow is: **load eval spec → discover tasks → create workspace → execute agent → collect events/usage → run graders → score/report results**.
   - This is orchestrated through `internal/orchestration`, `internal/execution`, `internal/trigger`, `internal/graders`, `internal/scoring`, `internal/metrics`, and `internal/reporting`.
   - The key output artifact is a structured **evaluation outcome JSON**, plus optional transcripts, JUnit, and human-readable summaries.

4. **Agent execution boundary**
   - `internal/execution` abstracts how a task is run against an AI agent engine.
   - There are two important execution modes:
     - a **mock/deterministic engine** for CI and local testing,
     - a **Copilot-backed engine** for real model execution.
   - The repository embeds platform-specific Copilot binaries under `internal/embedded/`, which is a significant architectural boundary: the CLI ships its own execution dependency rather than relying purely on user-installed tooling.

5. **Skills, eval specs, and workspace layout**
   - `internal/skill`, `internal/workspace`, `internal/projectconfig`, `internal/config`, and `internal/validation` manage the repository’s skill/eval conventions.
   - Skills are centered around `SKILL.md`; eval suites are YAML-based and can reference fixtures, tasks, hooks, and CSV-generated task rows.
   - The system supports discovery of skills/evals across directories, so diagrammatically this is a **filesystem-driven domain model** rather than a server-side database model.

6. **Task generation, discovery, and scaffolding**
   - `internal/scaffold`, `internal/wizard`, `internal/suggest`, and `cmd/waza/newtask` support creation of new skills, eval suites, and tasks.
   - `internal/discovery` and `internal/orchestration/skill_discovery.go` locate skills and related evals automatically.
   - This makes the repo act as both a **benchmark runner** and a **content authoring tool**.

7. **Grading and validation subsystem**
   - `internal/graders` contains the plugin-like grading implementations: text, file, diff, behavior, action sequence, skill invocation, prompt/LLM-as-judge, trigger tests, JSON schema, inline scripts, and program-based graders.
   - `internal/checks` and `internal/quality` provide preflight readiness and qualitative review of skills before run-time evaluation.
   - Graders operate over multiple inputs: agent output text, workspace file state, event traces, and sometimes judge-model responses.

8. **Telemetry, transcripts, and session observability**
   - `internal/copilotevents`, `internal/session`, `internal/transcript`, and `internal/metrics` capture execution history.
   - Session logs can be written as NDJSON and later viewed with `waza session view`.
   - These event streams are also what enable the dashboard, trend charts, and session digests in the UI.

9. **Results storage and comparison**
   - Results are stored locally and optionally uploaded to **Azure Blob Storage** via `internal/storage`.
   - `waza results` and `waza compare` operate on saved run artifacts, allowing historical comparison across runs and models.
   - This creates a clear separation between **execution time** and **analysis time**: the CLI can produce results once, then later compare, grade, or visualize them.

10. **Serving and UI layer**
   - `waza serve` starts an HTTP dashboard from `internal/webserver` and `internal/webapi`, backed by result files or stored runs.
   - There is also an optional **JSON-RPC server** path for IDE integration.
   - The `web/` directory is a separate React/Vite frontend that consumes the server API and renders dashboards, run detail views, trajectory diffs, trends, and live/session views.

11. **Configuration and schema boundaries**
   - `.waza.yaml`, eval YAML, task YAML, and JSON schemas under `schemas/` define the contract for the whole system.
   - `internal/config`, `internal/projectconfig`, `internal/validation`, and `schemas/embed.go` provide validation and compatibility checks.
   - These config files are not just settings; they are the primary integration surface for all subsystems.

12. **Token analysis as a first-class feature**
   - `cmd/waza/tokens` and `internal/tokens` implement markdown token counting, comparison, profiling, and suggestion workflows.
   - This is tightly connected to skill authoring because `SKILL.md` size and structure directly affect benchmark quality and submission readiness.
   - Token-aware checks feed into `waza check`, `waza dev`, and `waza suggest`.

13. **Supporting assets and examples**
   - `examples/` and `skills/` are not runtime code paths but are important architecture examples: they show how eval suites, custom agents, graders, and CI workflows are meant to be composed.
   - GitHub workflows in `.github/workflows/` and example CI files demonstrate that the intended deployment mode is **CLI-in-CI**, often with artifacts uploaded for later review.

14. **External/runtime dependencies that matter architecturally**
   - The repo integrates with **Copilot SDK/execution**, **Azure Blob Storage**, **GitHub Actions**, and optionally browser-based dashboard usage.
   - The Go CLI is the primary runtime; the React app and docs site are separate build artifacts, not part of the CLI execution path.
   - Architecture-wise, the repo is best understood as a **local-first evaluation platform** with optional cloud persistence and a separate visualization layer.

Main entry points: [`internal/orchestration`](https://github.com/microsoft/waza/tree/main/internal/orchestration) (Orchestration), [`schemas`](https://github.com/microsoft/waza/tree/main/schemas) (Config schema), [`internal/workspace`](https://github.com/microsoft/waza/tree/main/internal/workspace) (Workspace), [`internal/execution`](https://github.com/microsoft/waza/tree/main/internal/execution) (Execution)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_cli["CLI"]
  node_cmd_waza["waza<br/>Go CLI entrypoint"]
  node_cmd_run["Run<br/>eval command<br/>[cmd_run.go]"]
  node_cmd_manage["Manage cmds<br/>workspace commands"]
end

subgraph group_core["Core pipeline"]
  node_orchestration{{"Orchestration<br/>pipeline coordinator"}}
  node_execution{{"Execution<br/>agent runtime"}}
  node_embedded_copilot["Copilot bin<br/>embedded runtime"]
  node_trigger["Triggers<br/>task discovery"]
  node_graders{{"Graders<br/>grading plugins"}}
  node_scoring["Scoring<br/>score aggregation"]
  node_reporting["Reporting<br/>outcome formatters"]
end

subgraph group_content["Content model"]
  node_config_schema["Config schema<br/>validation boundary"]
  node_workspace["Workspace<br/>filesystem model"]
  node_skill_model["Skills<br/>skill model"]
  node_scaffold_tools["Scaffold<br/>authoring tools"]
end

subgraph group_serving["Serving and UI"]
  node_session_metrics["Session logs<br/>telemetry"]
  node_storage[("Storage<br/>artifact store")]
  node_webserver_api["Web API<br/>HTTP API"]
  node_web_ui["Dashboard<br/>React UI"]
end

node_cmd_waza -->|"dispatches"| node_cmd_run
node_cmd_waza -->|"hosts"| node_cmd_manage
node_cmd_run -->|"resolves"| node_workspace
node_cmd_manage -->|"validates"| node_config_schema
node_cmd_manage -->|"creates"| node_scaffold_tools
node_orchestration -->|"loads"| node_workspace
node_orchestration -->|"discovers"| node_trigger
node_orchestration -->|"runs"| node_execution
node_orchestration -->|"grades"| node_graders
node_orchestration -->|"scores"| node_scoring
node_orchestration -->|"reports"| node_reporting
node_execution -->|"uses"| node_embedded_copilot
node_execution -->|"emits"| node_session_metrics
node_trigger -->|"conforms to"| node_config_schema
node_graders -->|"reads"| node_config_schema
node_session_metrics -->|"persists"| node_storage
node_storage -->|"serves"| node_webserver_api
node_webserver_api -->|"feeds"| node_web_ui
node_skill_model -->|"discovered by"| node_workspace
node_scaffold_tools -->|"writes"| node_skill_model
node_config_schema -->|"governs"| node_workspace

click node_cmd_waza "https://github.com/microsoft/waza/tree/main/cmd/waza"
click node_cmd_run "https://github.com/microsoft/waza/blob/main/cmd/waza/cmd_run.go"
click node_orchestration "https://github.com/microsoft/waza/tree/main/internal/orchestration"
click node_execution "https://github.com/microsoft/waza/tree/main/internal/execution"
click node_embedded_copilot "https://github.com/microsoft/waza/tree/main/internal/embedded"
click node_trigger "https://github.com/microsoft/waza/tree/main/internal/trigger"
click node_graders "https://github.com/microsoft/waza/tree/main/internal/graders"
click node_scoring "https://github.com/microsoft/waza/tree/main/internal/scoring"
click node_reporting "https://github.com/microsoft/waza/tree/main/internal/reporting"
click node_session_metrics "https://github.com/microsoft/waza/tree/main/internal/session"
click node_storage "https://github.com/microsoft/waza/tree/main/internal/storage"
click node_webserver_api "https://github.com/microsoft/waza/tree/main/internal/webserver"
click node_web_ui "https://github.com/microsoft/waza/tree/main/web"
click node_config_schema "https://github.com/microsoft/waza/tree/main/schemas"
click node_workspace "https://github.com/microsoft/waza/tree/main/internal/workspace"
click node_skill_model "https://github.com/microsoft/waza/tree/main/internal/skill"
click node_scaffold_tools "https://github.com/microsoft/waza/tree/main/internal/scaffold"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cmd_waza,node_cmd_run,node_cmd_manage toneBlue
class node_orchestration,node_execution,node_embedded_copilot,node_trigger,node_graders,node_scoring,node_reporting toneAmber
class node_config_schema,node_workspace,node_skill_model,node_scaffold_tools toneMint
class node_session_metrics,node_storage,node_webserver_api,node_web_ui toneRose
```

## Components

### CLI

Command entrypoints and user-facing subcommands

- [waza](https://github.com/microsoft/waza/tree/main/cmd/waza): Go CLI entrypoint · Parses flags and dispatches subcommands; the main runtime boundary. · `cmd/waza`
- [Run](https://github.com/microsoft/waza/blob/main/cmd/waza/cmd_run.go): eval command · Executes eval suites and drives the main agent loop. · `cmd/waza/cmd_run.go`
- **Manage cmds**: workspace commands · Init, new, grade, check, suggest, compare, coverage, tokens, serve, results, dev, session.

### Core pipeline

Evaluation, execution, grading, scoring, and reporting

- [Orchestration](https://github.com/microsoft/waza/tree/main/internal/orchestration): pipeline coordinator · Coordinates eval discovery, workspace setup, execution, graders, and outcome assembly. · `internal/orchestration`
- [Execution](https://github.com/microsoft/waza/tree/main/internal/execution): agent runtime · Abstracts agent runs and supports mock and Copilot-backed engines. · `internal/execution`
- [Copilot bin](https://github.com/microsoft/waza/tree/main/internal/embedded): embedded runtime · Platform-specific embedded Copilot binaries shipped with the CLI. · `internal/embedded`
- [Triggers](https://github.com/microsoft/waza/tree/main/internal/trigger): task discovery · Discovers trigger specs and drives trigger-based evals. · `internal/trigger`
- [Graders](https://github.com/microsoft/waza/tree/main/internal/graders): grading plugins · Text, file, diff, behavior, action sequence, prompt, program, schema, and other grader types. · `internal/graders`
- [Scoring](https://github.com/microsoft/waza/tree/main/internal/scoring): score aggregation · Combines grader outputs into final scores and metrics. · `internal/scoring`
- [Reporting](https://github.com/microsoft/waza/tree/main/internal/reporting): outcome formatters · Produces evaluation outcome JSON, summaries, and JUnit artifacts. · `internal/reporting`

### Content model

Skills, eval specs, workspace conventions, and scaffolding

- [Config schema](https://github.com/microsoft/waza/tree/main/schemas): validation boundary · Defines and validates .waza.yaml, eval YAML, task YAML, and JSON schemas. · `schemas`
- [Workspace](https://github.com/microsoft/waza/tree/main/internal/workspace): filesystem model · Resolves project layout, skills, evals, and workspace conventions. · `internal/workspace`
- [Skills](https://github.com/microsoft/waza/tree/main/internal/skill): skill model · Represents SKILL.md-based skills and skill agents. · `internal/skill`
- [Scaffold](https://github.com/microsoft/waza/tree/main/internal/scaffold): authoring tools · Creates new skills, eval suites, and tasks; includes wizards and suggestion flows. · `internal/scaffold`

### Serving and UI

Local storage, APIs, dashboard, and docs frontend

- [Session logs](https://github.com/microsoft/waza/tree/main/internal/session): telemetry · Captures events, usage, transcripts, and session views for later analysis. · `internal/session`
- [Storage](https://github.com/microsoft/waza/tree/main/internal/storage): artifact store · Persists run artifacts locally and to Azure Blob Storage. · `internal/storage`
- [Web API](https://github.com/microsoft/waza/tree/main/internal/webserver): HTTP API · Serves dashboard data from stored runs and session artifacts. · `internal/webserver`
- [Dashboard](https://github.com/microsoft/waza/tree/main/web): React UI · Browser frontend for runs, trends, diffs, and live/session views. · `web`

## Connections

- waza → Run: dispatches · Main CLI routes into run/eval execution.
- waza → Manage cmds: hosts · Other subcommands share the same entrypoint.
- Run → Workspace: resolves · Finds project/workspace layout and eval locations.
- Manage cmds → Config schema: validates · Commands inspect and validate repository contracts.
- Manage cmds → Scaffold: creates · Init/new/suggest/wizard workflows author content.
- Orchestration → Workspace: loads · Discovers skills, evals, and task rows from the filesystem.
- Orchestration → Triggers: discovers · Builds trigger sets and task-specific execution plans.
- Orchestration → Execution: runs · Hands tasks to the agent runtime.
- Orchestration → Graders: grades · Executes plugin-like graders over outputs and traces.
- Orchestration → Scoring: scores · Aggregates grader results into final outcomes.
- Orchestration → Reporting: reports · Emits structured results and human-readable summaries.
- Execution → Copilot bin: uses · Copilot-backed execution relies on shipped binaries.
- Execution → Session logs: emits · Collects usage, events, and transcripts during runs.
- Triggers → Config schema: conforms to · Trigger and task definitions are schema-driven.
- Graders → Config schema: reads · Grader definitions and parameters are validated from config/specs.
- Session logs → Storage: persists · Session and run artifacts are stored for later review.
- Storage → Web API: serves · The dashboard API reads local or blob-backed artifacts.
- Web API → Dashboard: feeds · Frontend consumes API data for dashboards and details.
- Skills → Workspace: discovered by · Skills are found through workspace scanning and conventions.
- Scaffold → Skills: writes · Scaffolding creates or updates skills and eval content.
- Config schema → Workspace: governs · Workspace and config files define the integration surface.
