How pantheon-org/skill-quality-auditor works
1. Repository purpose
- This is a Go-based CLI for auditing “skills” represented as SKILL.md files.
- Its core job is to score a skill against a 9-dimension rubric, detect duplication, generate remediation guidance, and validate supporting artifacts/evals.
2. Top-level execution path
- main.go and cmd/root.go form the CLI entrypoint.
- Subcommands under cmd/ implement the user-facing workflows: evaluate, batch, analyze, duplication, aggregate, remediate, trend, validate, init, update, and prune.
- The CLI is the orchestration layer; most other packages are domain engines or output adapters.
3. Skill evaluation pipeline
- scorer/ contains the D1–D9 dimension logic, grade thresholds, and shared scoring utilities.
- Input is a skill directory or SKILL.md; output is a structured score, grade, and warnings tied to specific dimensions.
- The scoring layer is intentionally split by dimension so each criterion has its own boundary and can evolve independently.
4. Analysis and signal extraction
- analysis/ provides TF-IDF keyword extraction and pattern detectors used to characterize a skill beyond its raw score.
- This layer feeds commands like analyze and also supports reporting by identifying structural signals, repeated concepts, and notable patterns.
- internal/tokenize/ is the low-level text normalization boundary used by analysis and similarity logic.
5. Duplication detection subsystem
- duplication/ computes similarity between skills using Jaccard-style overlap and inventory/similarity helpers.
- It powers the duplication command and supports family-level consolidation analysis.
- This subsystem is separate from scoring: it answers “how much overlap exists?” rather than “how good is this skill?”
6. Reporting and persistence
- reporter/ converts raw scoring/analysis results into human-readable or machine-readable outputs.
- It also handles storing audit snapshots and generating remediation plans.
- trend depends on persisted audit history to compare score deltas over time, so the reporting layer is also the repository’s audit archive interface.
7. Validation and quality gates
- cmd/validate.go plus reporter validation helpers check repository conventions and artifact/report structure.
- This is the quality-control boundary for generated content, especially review reports and remediation plans.
- CI uses these checks as gates, so invalid artifact structure is treated as a release-quality issue, not just a formatting problem.
8. Agent/environment bootstrapping
- agents/registry.go defines supported agent targets for the init command.
- This subsystem is responsible for installing or registering the auditor skill into external agent environments, making it a deployment/onboarding boundary rather than part of scoring.
9. Embedded assets and specs
- cmd/assets/ and .agents/skills/ contain the bundled skill content, schemas, templates, eval scenarios, and reference docs used at runtime.
- These assets are effectively the product’s embedded specification: they define expected file shapes, evaluation scenarios, remediation templates, and command behavior references.
- docs/ADR/ records architectural decisions that explain why major boundaries exist and should be treated as the design history.
10. External service boundary
- internal/llmclient/ abstracts provider-specific clients for OpenAI, Anthropic, and Gemini behind a shared interface.
- This is an optional integration boundary for any LLM-assisted features, insulated from the main scoring engine so provider changes do not leak into core CLI logic.
11. Automation and release workflow
- .github/workflows/ defines CI, release, and skill-quality checks; scripts/ contains install and maintenance scripts.
- goreleaser, release-please, and the install script support distribution of the CLI binary, while CI validates tests, linting, and repository quality.
- The architecture therefore includes a build/release pipeline as a first-class system boundary, not just a developer convenience.
12. Test and fixture model
- testdata/fixtures/ provides representative skill directories used to test the evaluator, analyzer, and validator behavior.
- Tests are organized around the same subsystem boundaries as production code, which suggests the repo is structured around end-to-end command behavior plus focused unit logic.
13. How the pieces fit together
- Typical flow: the CLI loads a skill, the scorer computes dimension scores, analysis and duplication engines add diagnostics, and reporter formats/stores the result.
- Optional flows: trend reads stored audits, remediate generates fix guidance, aggregate reasons about a skill family, and validate checks output artifacts.
- The main architectural separation is between command orchestration, domain engines, reporting/persistence, and embedded specification/assets.
Main entry points: cmd workflows · reporter reporter · analysis analysis · duplication duplication
Components 18
CLI orchestration
Entrypoint and user-facing commands that coordinate repository workflows.
Core engines
Domain logic for scoring, analysis, duplication, and validation.
- scorer · Dimension scoring, thresholds, grades, and shared scoring utilities.
scorer - analysis · Keyword extraction and pattern detection for structural signals beyond raw score.
analysis - tokenize · Low-level tokenization and normalization used by analysis and similarity logic.
internal/tokenize - duplication · Jaccard-style overlap, inventory, and similarity helpers for skill duplication.
duplication
Reporting and audit
Formatting, persistence, remediation, and trend/history access.
- reporter · Turns scores and analysis into reports, remediation plans, and persisted audits.
reporter - audit store · Persistence boundary for saved audits and trend comparisons.
reporter/store.go - remediation · Creates remediation guidance and structured fix plans.
reporter/remediation.go - validate · Checks repository conventions and artifact/report structure.
cmd/validate.go
Integrations
External runtime boundaries and environment bootstrap logic.
- agents · Registers supported agent targets and installs the auditor skill into external environments.
agents - llm client · Provider-agnostic wrapper over OpenAI, Anthropic, and Gemini clients.
internal/llmclient
Embedded spec
Bundled assets, reference material, and design history that define expected behavior.
- cmd assets · Embedded schemas, templates, references, evals, and command asset docs used at runtime.
cmd/assets - skill specs · Bundled SKILL.md content, schemas, templates, eval scenarios, and references for agent skills.
.agents/skills - ADR docs · Architecture decision records explaining major boundaries and policies.
docs/ADR
Delivery pipeline
Automation, scripts, and release/CI workflow boundary.
- CI workflows · GitHub Actions, release tooling, and quality checks for build and publication.
.github/workflows - scripts · Install and maintenance scripts supporting setup and release workflows.
scripts
Connections 23
- main.go → root cmd: starts · Main hands execution to the root CLI command.
- root cmd → workflows: dispatches · Root command exposes the user-facing subcommands.
- workflows → scorer: scores · Evaluate and batch commands invoke the dimension scoring engine.
- workflows → analysis: analyzes · Analyze and reporting-oriented commands use signal extraction.
- workflows → duplication: compares · Duplication and aggregate workflows use similarity detection.
- workflows → reporter: formats · Commands pass results into report generation and remediation output.
- workflows → validate: checks · Validate command gates generated artifacts and repository conventions.
- scorer → tokenize: normalizes · Scoring logic shares token/text normalization helpers where needed.
- analysis → tokenize: tokenizes · Pattern and TF-IDF analysis consume normalized tokens.
- duplication → tokenize: tokenizes · Similarity logic relies on shared tokenization for overlap comparison.
- reporter → audit store: persists · Reporter stores audit snapshots and serves trend history.
- reporter → remediation: generates · Reporter produces remediation plans from audit findings.
- analysis → reporter: feeds · Analysis outputs are incorporated into human-readable reports.
- duplication → reporter: feeds · Duplication findings are rendered through the reporting layer.
- validate → reporter: validates · Validation helpers check report and remediation structure.
- agents → workflows: supports init · Agent registry powers onboarding and installation workflows.
- llm client → workflows: optional · LLM-backed features are isolated behind a provider adapter boundary.
- cmd assets → workflows: supplies · Embedded command assets define schemas, templates, and reference docs.
- skill specs → agents: installs · Bundled skill content and specs support agent environment bootstrap.
- ADR docs → scorer: documents · ADR history explains scoring dimension and threshold design choices.
- ADR docs → reporter: documents · ADR history captures reporting and remediation architecture decisions.
- CI workflows → validate: gates · CI uses validation and quality checks as release gates.
- CI workflows → scripts: runs · Automation and release jobs rely on maintenance scripts.