How vadiml1024/semcheck works
Project type and purpose
• Semcheck is a command‐line and server‐based tool in Go.
• Purpose: Validate that code matches specifications by running semantic rules through LLM providers.
• Modes of operation:
– CLI mode (cmd/cli) for local execution and pre‐commit hooks
– Evaluation mode (cmd/eval) for running test cases and benchmarks
– MCP Server mode (internal/mcp, main.go -mcp-server) for TCP‐based integration.
File structure
• Top‐level “cmd” directory contains the entrypoints:
– cmd/cli → CLI commands (init, default check)
– cmd/eval → test harness for evaluation cases.
• main.go ties together config parsing and mode selection.
• internal/ houses core packages:
– config: loads and validates semcheck.yaml
– checker: orchestrates rules, invokes prompts, collects LLM responses, reporters
– processor: matches files to rules, reads specs and implementations
– providers: pluggable LLM client interfaces (OpenAI, Anthropic, Gemini, Cerebras, Ollama)
– mcp: TCP server implementation of Model Context Protocol
• Other directories:
– docs/, specs/: human‐readable specifications and protocol docs
– evals/ and examples/: sample inputs, test cases, and example configs.
Extract architectural clues from the README
• Key components mentioned: CLI, “Bring Your Own Model” providers, MCP server, remote specs.
• Workflow: config → rule dispatch → LLM provider → reporter (stdout, GitHub).
• Technologies: Go modules, Just, GitHub Actions integration.
Main components & their interactions
A. Frontend / Entry Points
1. CLI Component (cmd/cli)
– Reads user flags, invokes init or check commands
2. Eval Component (cmd/eval)
– Runs predefined evaluation cases for development
B. Core Engine
3. Config Loader (internal/config)
– Parses semcheck.yaml or passed flags
4. Processor (internal/processor)
– Scans filesystem, matches files to rules
– Reads spec and implementation content
5. Checker (internal/checker)
– Builds prompt templates, calls LLM client
– Aggregates LLM responses, applies pass/fail logic
6. Reporter (internal/checker/*.go)
– Outputs results to stdout or GitHub via reporter interfaces
C. External Integrations
7. Providers (internal/providers)
– Abstract client interface
– Implementations for OpenAI, Anthropic, Gemini, Cerebras, Ollama
8. Remote Spec Fetcher (HTTP)
– Fetches spec text from URLs
9. MCP Server (internal/mcp)
– Listens on TCP port, translates MCP requests to Checker calls
D. Supporting Elements
10. Examples & Docs (examples, docs, specs)
– Input templates, documentation
11. Test Suites (internal/*_test.go, evals/cases)
– Validation of each component’s behavior
E. External Services
12. LLM APIs (OpenAI, Anthropic, etc.)
13. GitHub (for GitHub Actions and GitHub Reporter)
• Config Loader → Processor → Checker
• Checker ↔ Providers (LLM request/response)
• Checker → Reporter → Stdout or GitHub API
• CLI with -mcp-server option starts MCP Server, which → Checker
• Processor may fetch spec files via HTTP → Remote Spec Server
• GitHub Actions workflow (external) → invokes CLI container
Architectural patterns & design principles
• Separation of Concerns: distinct packages for config, processing, checking, reporting
• Plugin pattern for LLM providers (common client.go interface)
• Hexagonal architecture for core engine → adapters (providers, reporters, MCP)
• Test‐driven development: extensive *_test.go files in each package
• Self‐hosted introspection: Semcheck checks its own specs (specs/semcheck.md)