How microsoft/waza works
1. Repository shape and purpose
- This is primarily a Go CLI product for building and evaluating AI agent skills.
- The main runtime is cmd/waza, supported by a large internal/ library layer.
- The repo also contains a browser dashboard (web/) and a documentation site (site/), plus reusable example eval suites and skills.
2. Top-level execution boundary
- The CLI is the entrypoint for almost everything: init, new, run, grade, check, suggest, compare, coverage, tokens, serve, results, dev, and session.
- cmd/waza parses flags and dispatches into internal packages; it does not hold business logic itself.
- waza can be used both in a project workspace with skills/ + evals/, or in standalone skill directories.
3. Evaluation pipeline is the core subsystem
- The central flow is: load eval spec → discover tasks → create workspace → execute agent → collect events/usage → run graders → score/report results.
- This is orchestrated through internal/orchestration, internal/execution, internal/trigger, internal/graders, internal/scoring, internal/metrics, and internal/reporting.
- The key output artifact is a structured evaluation outcome JSON, plus optional transcripts, JUnit, and human-readable summaries.
4. Agent execution boundary
- internal/execution abstracts how a task is run against an AI agent engine.
- There are two important execution modes:
- a mock/deterministic engine for CI and local testing,
- a Copilot-backed engine for real model execution.
- The repository embeds platform-specific Copilot binaries under internal/embedded/, which is a significant architectural boundary: the CLI ships its own execution dependency rather than relying purely on user-installed tooling.
5. Skills, eval specs, and workspace layout
- internal/skill, internal/workspace, internal/projectconfig, internal/config, and internal/validation manage the repository’s skill/eval conventions.
- Skills are centered around SKILL.md; eval suites are YAML-based and can reference fixtures, tasks, hooks, and CSV-generated task rows.
- The system supports discovery of skills/evals across directories, so diagrammatically this is a filesystem-driven domain model rather than a server-side database model.
6. Task generation, discovery, and scaffolding
- internal/scaffold, internal/wizard, internal/suggest, and cmd/waza/newtask support creation of new skills, eval suites, and tasks.
- internal/discovery and internal/orchestration/skill_discovery.go locate skills and related evals automatically.
- This makes the repo act as both a benchmark runner and a content authoring tool.
7. Grading and validation subsystem
- internal/graders contains the plugin-like grading implementations: text, file, diff, behavior, action sequence, skill invocation, prompt/LLM-as-judge, trigger tests, JSON schema, inline scripts, and program-based graders.
- internal/checks and internal/quality provide preflight readiness and qualitative review of skills before run-time evaluation.
- Graders operate over multiple inputs: agent output text, workspace file state, event traces, and sometimes judge-model responses.
8. Telemetry, transcripts, and session observability
- internal/copilotevents, internal/session, internal/transcript, and internal/metrics capture execution history.
- Session logs can be written as NDJSON and later viewed with waza session view.
- These event streams are also what enable the dashboard, trend charts, and session digests in the UI.
9. Results storage and comparison
- Results are stored locally and optionally uploaded to Azure Blob Storage via internal/storage.
- waza results and waza compare operate on saved run artifacts, allowing historical comparison across runs and models.
- This creates a clear separation between execution time and analysis time: the CLI can produce results once, then later compare, grade, or visualize them.
10. Serving and UI layer
- waza serve starts an HTTP dashboard from internal/webserver and internal/webapi, backed by result files or stored runs.
- There is also an optional JSON-RPC server path for IDE integration.
- The web/ directory is a separate React/Vite frontend that consumes the server API and renders dashboards, run detail views, trajectory diffs, trends, and live/session views.
11. Configuration and schema boundaries
- .waza.yaml, eval YAML, task YAML, and JSON schemas under schemas/ define the contract for the whole system.
- internal/config, internal/projectconfig, internal/validation, and schemas/embed.go provide validation and compatibility checks.
- These config files are not just settings; they are the primary integration surface for all subsystems.
12. Token analysis as a first-class feature
- cmd/waza/tokens and internal/tokens implement markdown token counting, comparison, profiling, and suggestion workflows.
- This is tightly connected to skill authoring because SKILL.md size and structure directly affect benchmark quality and submission readiness.
- Token-aware checks feed into waza check, waza dev, and waza suggest.
13. Supporting assets and examples
- examples/ and skills/ are not runtime code paths but are important architecture examples: they show how eval suites, custom agents, graders, and CI workflows are meant to be composed.
- GitHub workflows in .github/workflows/ and example CI files demonstrate that the intended deployment mode is CLI-in-CI, often with artifacts uploaded for later review.
14. External/runtime dependencies that matter architecturally
- The repo integrates with Copilot SDK/execution, Azure Blob Storage, GitHub Actions, and optionally browser-based dashboard usage.
- The Go CLI is the primary runtime; the React app and docs site are separate build artifacts, not part of the CLI execution path.
- Architecture-wise, the repo is best understood as a local-first evaluation platform with optional cloud persistence and a separate visualization layer.
Main entry points: orchestration Orchestration · schemas Config schema · workspace Workspace · execution Execution
Components 18
CLI
Command entrypoints and user-facing subcommands
Core pipeline
Evaluation, execution, grading, scoring, and reporting
- Orchestration · Coordinates eval discovery, workspace setup, execution, graders, and outcome assembly.
internal/orchestration - Execution · Abstracts agent runs and supports mock and Copilot-backed engines.
internal/execution - Copilot bin · Platform-specific embedded Copilot binaries shipped with the CLI.
internal/embedded - Triggers · Discovers trigger specs and drives trigger-based evals.
internal/trigger - Graders · Text, file, diff, behavior, action sequence, prompt, program, schema, and other grader types.
internal/graders - Scoring · Combines grader outputs into final scores and metrics.
internal/scoring - Reporting · Produces evaluation outcome JSON, summaries, and JUnit artifacts.
internal/reporting
Content model
Skills, eval specs, workspace conventions, and scaffolding
- Config schema · Defines and validates .waza.yaml, eval YAML, task YAML, and JSON schemas.
schemas - Workspace · Resolves project layout, skills, evals, and workspace conventions.
internal/workspace - Skills · Represents SKILL.md-based skills and skill agents.
internal/skill - Scaffold · Creates new skills, eval suites, and tasks; includes wizards and suggestion flows.
internal/scaffold
Serving and UI
Local storage, APIs, dashboard, and docs frontend
- Session logs · Captures events, usage, transcripts, and session views for later analysis.
internal/session - Storage · Persists run artifacts locally and to Azure Blob Storage.
internal/storage - Web API · Serves dashboard data from stored runs and session artifacts.
internal/webserver - Dashboard · Browser frontend for runs, trends, diffs, and live/session views.
web
Connections 21
- waza → Run: dispatches · Main CLI routes into run/eval execution.
- waza → Manage cmds: hosts · Other subcommands share the same entrypoint.
- Run → Workspace: resolves · Finds project/workspace layout and eval locations.
- Manage cmds → Config schema: validates · Commands inspect and validate repository contracts.
- Manage cmds → Scaffold: creates · Init/new/suggest/wizard workflows author content.
- Orchestration → Workspace: loads · Discovers skills, evals, and task rows from the filesystem.
- Orchestration → Triggers: discovers · Builds trigger sets and task-specific execution plans.
- Orchestration → Execution: runs · Hands tasks to the agent runtime.
- Orchestration → Graders: grades · Executes plugin-like graders over outputs and traces.
- Orchestration → Scoring: scores · Aggregates grader results into final outcomes.
- Orchestration → Reporting: reports · Emits structured results and human-readable summaries.
- Execution → Copilot bin: uses · Copilot-backed execution relies on shipped binaries.
- Execution → Session logs: emits · Collects usage, events, and transcripts during runs.
- Triggers → Config schema: conforms to · Trigger and task definitions are schema-driven.
- Graders → Config schema: reads · Grader definitions and parameters are validated from config/specs.
- Session logs → Storage: persists · Session and run artifacts are stored for later review.
- Storage → Web API: serves · The dashboard API reads local or blob-backed artifacts.
- Web API → Dashboard: feeds · Frontend consumes API data for dashboards and details.
- Skills → Workspace: discovered by · Skills are found through workspace scanning and conventions.
- Scaffold → Skills: writes · Scaffolding creates or updates skills and eval content.
- Config schema → Workspace: governs · Workspace and config files define the integration surface.