How nvidia/skillevaluator works
SkillEvaluator evaluates agent-skill directories through three independent tiers: static validation, guidance deduplication, and live agent runs, then presents findings and run evidence. The README documents the user-facing commands and their outputs; sampled source confirms selected Tier 3 orchestration, Harbor result collection, inference, and report-rendering relationships. Some tier internals are represented at their source directories because their implementations were not sampled; unsupported calls are omitted.
Main entry points: tier2/commands.py Tier 2 deduplication · tier3/commands.py Tier 3 orchestration · cli.py CLI commands · tier1/commands.py Tier 1 checks
Components 25
User entry points
- CLI commands
src/skillevaluator/cli.py - Evaluation service
src/skillevaluator/evaluation/service.py
Static validation
- Tier 1 checks
src/skillevaluator/tier1/commands.py - Quality and safety validators
src/skillevaluator/validators
Deduplication
- Tier 2 deduplication
src/skillevaluator/tier2/commands.py - Intra-skill analysis
src/skillevaluator/deduplication/intra_skill/llm_analyzer.py - Plugin deduplication
src/skillevaluator/deduplication/plugin/llm_analyzer.py - Embedding client
src/skillevaluator/embedding/client.py - Skill catalog collection
src/skillevaluator/deduplication/utils/skill_collector.py
Live evaluation
- Tier 3 orchestration
src/skillevaluator/tier3/commands.py - Evaluation datasets
src/skillevaluator/tier3/generate_dataset.py - Harbor runner
src/skillevaluator/tier3/harbor/runner.py - Run result collector
src/skillevaluator/tier3/harbor/collector.py - Trajectory and task checks
src/skillevaluator/tier3/eval_core/checks.py - Tool-call normalizer
src/skillevaluator/tier3/eval_core/codex_tool_call_normalizer.py - LLM judge
src/skillevaluator/tier3/eval_core/llm_judge.py
Results and reports
- Evaluation results
src/skillevaluator/models/result.py - HTML reports
src/skillevaluator/reporting/html.py - Retained reports
src/skillevaluator/reporting/retained.py - Repository CI reports
ci/run_tier1.py - Report views
ci/report_views.py - Report publisher
ci/publish_reports.py
Other components
- Skill author
- Supported agent runtime
- Model provider
Connections 25
- Skill author → CLI commands: runs command
README.md - CLI commands → Tier 1 checks: runs checks
- CLI commands → Tier 2 deduplication: runs deduplication
- CLI commands → Evaluation service: starts evaluation
- Tier 1 checks → Quality and safety validators: validates skills
- Tier 2 deduplication → Intra-skill analysis: analyzes guidance
- Tier 2 deduplication → Plugin deduplication: checks plugin overlap
- Tier 2 deduplication → Embedding client: uses embeddings
- Tier 2 deduplication → Skill catalog collection: compares catalog
- Evaluation service → Tier 3 orchestration: delegates evaluation
src/skillevaluator/evaluation/service.py - Tier 3 orchestration → Evaluation datasets: generates cases
src/skillevaluator/tier3/commands.py - Tier 3 orchestration → Harbor runner: runs jobs
src/skillevaluator/tier3/commands.py - Harbor runner → Supported agent runtime: executes tasks
- Harbor runner → Run result collector: collects outputs
- Run result collector → Trajectory and task checks: uses checks
src/skillevaluator/tier3/harbor/collector.py - Run result collector → Tool-call normalizer: normalizes calls
src/skillevaluator/tier3/harbor/collector.py - LLM judge → Model provider: requests inference
- Tier 3 orchestration → LLM judge: judges runs
- Tier 3 orchestration → Evaluation results: produces results
- Evaluation results → HTML reports: renders report
- Tier 1 checks → Repository CI reports: runs CI checks
README.md - Repository CI reports → Retained reports: retains evidence
ci/run_tier1.py - Report publisher → Retained reports: reads evidence
ci/publish_reports.py - Report publisher → Report views: renders views
ci/publish_reports.py - Report views → HTML reports: uses report renderer
ci/report_views.py