# nvidia/skillevaluator architecture

> Architecture diagram of the GitHub repository nvidia/skillevaluator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvidia/skillevaluator
- Repository: https://github.com/nvidia/skillevaluator

## Overview

SkillEvaluator evaluates agent-skill directories through three independent tiers: static validation, guidance deduplication, and live agent runs, then presents findings and run evidence. The README documents the user-facing commands and their outputs; sampled source confirms selected Tier 3 orchestration, Harbor result collection, inference, and report-rendering relationships. Some tier internals are represented at their source directories because their implementations were not sampled; unsupported calls are omitted.

Main entry points: [`src/skillevaluator/tier2/commands.py`](https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/tier2/commands.py) (Tier 2 deduplication), [`src/skillevaluator/tier3/commands.py`](https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/tier3/commands.py) (Tier 3 orchestration), [`src/skillevaluator/cli.py`](https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/cli.py) (CLI commands), [`src/skillevaluator/tier1/commands.py`](https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/tier1/commands.py) (Tier 1 checks)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["User entry points"]
  node_cli["CLI commands<br/>[cli.py]"]
  node_evalservice["Evaluation service<br/>[service.py]"]
end

subgraph group_validation["Static validation"]
  node_tier1["Tier 1 checks<br/>[commands.py]"]
  node_validators["Quality and safety validators"]
end

subgraph group_dedup["Deduplication"]
  node_tier2["Tier 2 deduplication<br/>[commands.py]"]
  node_skilldedup["Intra-skill analysis<br/>[llm_analyzer.py]"]
  node_plugindup["Plugin deduplication<br/>[llm_analyzer.py]"]
  node_embedding["Embedding client<br/>[client.py]"]
  node_catalog["Skill catalog collection<br/>[skill_collector.py]"]
end

subgraph group_live["Live evaluation"]
  node_tier3["Tier 3 orchestration<br/>[commands.py]"]
  node_dataset["Evaluation datasets"]
  node_harbor["Harbor runner<br/>[runner.py]"]
  node_collector["Run result collector<br/>[collector.py]"]
  node_evalchecks["Trajectory and task checks<br/>[checks.py]"]
  node_normalizer["Tool-call normalizer"]
  node_llmjudge["LLM judge<br/>[llm_judge.py]"]
end

subgraph group_reporting["Results and reports"]
  node_result["Evaluation results<br/>[result.py]"]
  node_report["HTML reports<br/>[html.py]"]
  node_retained["Retained reports<br/>[retained.py]"]
  node_ci["Repository CI reports<br/>[run_tier1.py]"]
  node_views["Report views<br/>[report_views.py]"]
  node_publisher["Report publisher<br/>[publish_reports.py]"]
end

node_user(("Skill author"))
node_agent{{"Supported agent runtime"}}
node_provider{{"Model provider"}}

node_user -->|"runs command"| node_cli
node_cli -.->|"runs checks"| node_tier1
node_cli -.->|"runs deduplication"| node_tier2
node_cli -.->|"starts evaluation"| node_evalservice
node_tier1 -.->|"validates skills"| node_validators
node_tier2 -.->|"analyzes guidance"| node_skilldedup
node_tier2 -.->|"checks plugin overlap"| node_plugindup
node_tier2 -.->|"uses embeddings"| node_embedding
node_tier2 -.->|"compares catalog"| node_catalog
node_evalservice -->|"delegates evaluation"| node_tier3
node_tier3 -->|"generates cases"| node_dataset
node_tier3 -->|"runs jobs"| node_harbor
node_harbor -.->|"executes tasks"| node_agent
node_harbor -.->|"collects outputs"| node_collector
node_collector -->|"uses checks"| node_evalchecks
node_collector -->|"normalizes calls"| node_normalizer
node_llmjudge -.->|"requests inference"| node_provider
node_tier3 -.->|"judges runs"| node_llmjudge
node_tier3 -.->|"produces results"| node_result
node_result -.->|"renders report"| node_report
node_tier1 -->|"runs CI checks"| node_ci
node_ci -->|"retains evidence"| node_retained
node_publisher -->|"reads evidence"| node_retained
node_publisher -->|"renders views"| node_views
node_views -->|"uses report renderer"| node_report

click node_cli "https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/cli.py"
click node_tier1 "https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/tier1/commands.py"
click node_validators "https://github.com/nvidia/skillevaluator/tree/main/src/skillevaluator/validators"
click node_tier2 "https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/tier2/commands.py"
click node_skilldedup "https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/deduplication/intra_skill/llm_analyzer.py"
click node_plugindup "https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/deduplication/plugin/llm_analyzer.py"
click node_embedding "https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/embedding/client.py"
click node_catalog "https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/deduplication/utils/skill_collector.py"
click node_evalservice "https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/evaluation/service.py"
click node_tier3 "https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/tier3/commands.py"
click node_dataset "https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/tier3/generate_dataset.py"
click node_harbor "https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/tier3/harbor/runner.py"
click node_collector "https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/tier3/harbor/collector.py"
click node_evalchecks "https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/tier3/eval_core/checks.py"
click node_normalizer "https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/tier3/eval_core/codex_tool_call_normalizer.py"
click node_llmjudge "https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/tier3/eval_core/llm_judge.py"
click node_result "https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/models/result.py"
click node_report "https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/reporting/html.py"
click node_retained "https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/reporting/retained.py"
click node_ci "https://github.com/nvidia/skillevaluator/blob/main/ci/run_tier1.py"
click node_views "https://github.com/nvidia/skillevaluator/blob/main/ci/report_views.py"
click node_publisher "https://github.com/nvidia/skillevaluator/blob/main/ci/publish_reports.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_evalservice toneBlue
class node_tier1,node_validators toneAmber
class node_tier2,node_skilldedup,node_plugindup,node_embedding,node_catalog toneMint
class node_tier3,node_dataset,node_harbor,node_collector,node_evalchecks,node_normalizer,node_llmjudge toneRose
class node_result,node_report,node_retained,node_ci,node_views,node_publisher,node_user,node_agent,node_provider toneIndigo
```

## Components

### User entry points

- [CLI commands](https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/cli.py): `src/skillevaluator/cli.py`
- [Evaluation service](https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/evaluation/service.py): `src/skillevaluator/evaluation/service.py`

### Static validation

- [Tier 1 checks](https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/tier1/commands.py): `src/skillevaluator/tier1/commands.py`
- [Quality and safety validators](https://github.com/nvidia/skillevaluator/tree/main/src/skillevaluator/validators): `src/skillevaluator/validators`

### Deduplication

- [Tier 2 deduplication](https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/tier2/commands.py): `src/skillevaluator/tier2/commands.py`
- [Intra-skill analysis](https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/deduplication/intra_skill/llm_analyzer.py): `src/skillevaluator/deduplication/intra_skill/llm_analyzer.py`
- [Plugin deduplication](https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/deduplication/plugin/llm_analyzer.py): `src/skillevaluator/deduplication/plugin/llm_analyzer.py`
- [Embedding client](https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/embedding/client.py): `src/skillevaluator/embedding/client.py`
- [Skill catalog collection](https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/deduplication/utils/skill_collector.py): `src/skillevaluator/deduplication/utils/skill_collector.py`

### Live evaluation

- [Tier 3 orchestration](https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/tier3/commands.py): `src/skillevaluator/tier3/commands.py`
- [Evaluation datasets](https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/tier3/generate_dataset.py): `src/skillevaluator/tier3/generate_dataset.py`
- [Harbor runner](https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/tier3/harbor/runner.py): `src/skillevaluator/tier3/harbor/runner.py`
- [Run result collector](https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/tier3/harbor/collector.py): `src/skillevaluator/tier3/harbor/collector.py`
- [Trajectory and task checks](https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/tier3/eval_core/checks.py): `src/skillevaluator/tier3/eval_core/checks.py`
- [Tool-call normalizer](https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/tier3/eval_core/codex_tool_call_normalizer.py): `src/skillevaluator/tier3/eval_core/codex_tool_call_normalizer.py`
- [LLM judge](https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/tier3/eval_core/llm_judge.py): `src/skillevaluator/tier3/eval_core/llm_judge.py`

### Results and reports

- [Evaluation results](https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/models/result.py): `src/skillevaluator/models/result.py`
- [HTML reports](https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/reporting/html.py): `src/skillevaluator/reporting/html.py`
- [Retained reports](https://github.com/nvidia/skillevaluator/blob/main/src/skillevaluator/reporting/retained.py): `src/skillevaluator/reporting/retained.py`
- [Repository CI reports](https://github.com/nvidia/skillevaluator/blob/main/ci/run_tier1.py): `ci/run_tier1.py`
- [Report views](https://github.com/nvidia/skillevaluator/blob/main/ci/report_views.py): `ci/report_views.py`
- [Report publisher](https://github.com/nvidia/skillevaluator/blob/main/ci/publish_reports.py): `ci/publish_reports.py`

### Other components

- **Skill author**
- **Supported agent runtime**
- **Model provider**

## Connections

- Skill author → CLI commands: runs command (evidence: `README.md`)
- CLI commands → Tier 1 checks: runs checks
- CLI commands → Tier 2 deduplication: runs deduplication
- CLI commands → Evaluation service: starts evaluation
- Tier 1 checks → Quality and safety validators: validates skills
- Tier 2 deduplication → Intra-skill analysis: analyzes guidance
- Tier 2 deduplication → Plugin deduplication: checks plugin overlap
- Tier 2 deduplication → Embedding client: uses embeddings
- Tier 2 deduplication → Skill catalog collection: compares catalog
- Evaluation service → Tier 3 orchestration: delegates evaluation (evidence: `src/skillevaluator/evaluation/service.py`)
- Tier 3 orchestration → Evaluation datasets: generates cases (evidence: `src/skillevaluator/tier3/commands.py`)
- Tier 3 orchestration → Harbor runner: runs jobs (evidence: `src/skillevaluator/tier3/commands.py`)
- Harbor runner → Supported agent runtime: executes tasks
- Harbor runner → Run result collector: collects outputs
- Run result collector → Trajectory and task checks: uses checks (evidence: `src/skillevaluator/tier3/harbor/collector.py`)
- Run result collector → Tool-call normalizer: normalizes calls (evidence: `src/skillevaluator/tier3/harbor/collector.py`)
- LLM judge → Model provider: requests inference
- Tier 3 orchestration → LLM judge: judges runs
- Tier 3 orchestration → Evaluation results: produces results
- Evaluation results → HTML reports: renders report
- Tier 1 checks → Repository CI reports: runs CI checks (evidence: `README.md`)
- Repository CI reports → Retained reports: retains evidence (evidence: `ci/run_tier1.py`)
- Report publisher → Retained reports: reads evidence (evidence: `ci/publish_reports.py`)
- Report publisher → Report views: renders views (evidence: `ci/publish_reports.py`)
- Report views → HTML reports: uses report renderer (evidence: `ci/report_views.py`)