How vkr-vavilla/llm_evaluation works
This repository is a two-part evaluation pipeline: a small Python inference service and a Go-based evaluator that benchmarks that service against a fixed ground-truth corpus. The overall architecture is designed for repeatable QA of LLM-generated WAV exploit payloads.
1. Top-level split: service vs. evaluator
- python_app/ is the model-facing API layer.
- evaluator/ is the benchmarking and reporting engine.
- data/ground_truth.json is the stable input dataset both CI and local runs consume.
- scripts/visualize.py is a post-processing/reporting tool, not part of the evaluation path itself.
2. Python inference boundary
- python_app/app.py exposes a /generate endpoint.
- It can run in mock mode for CI/local testing or with a real LLM backend such as CodeLlama.
- This is the main external boundary the evaluator talks to: it receives prompts and returns hex-encoded WAV payloads.
3. Payload validation layer
- python_app/validator.py enforces RIFF/WAVE header correctness on generated outputs.
- This creates an early correctness gate before results are scored, so the system tracks both semantic similarity and basic file-structure validity.
4. Go evaluation engine
- evaluator/main.go orchestrates the benchmark run: it loads the dataset, sends prompts to the Python API, gathers responses, computes metrics, and emits results.
- The evaluator is the control plane for the whole workflow and likely the component that writes results.csv and the scorecard mentioned in the README.
5. Metric computation subsystem
- evaluator/metrics.go contains the scoring logic: normalized Levenshtein similarity, latency percentiles, and standard deviation calculations.
- evaluator/metrics_test.go provides unit coverage for these computations, which is important because the evaluation output is only as trustworthy as the metric code.
6. Data model boundary
- evaluator/types.go defines the shared structures used by the evaluator for dataset entries, responses, and result rows.
- This keeps the run logic decoupled from the dataset format and output format, which matters for diagramming the data flow cleanly.
7. Dataset role
- data/ground_truth.json is the canonical benchmark corpus: crash-inducing payloads plus valid/edge-case WAVs.
- The evaluator treats this as read-only input, so the repository has a strong separation between immutable benchmark data and generated outputs.
8. Execution flow
- A run starts with the evaluator reading ground_truth.json.
- For each case, it sends a prompt to the Python /generate endpoint.
- The response is validated and scored against ground truth.
- Aggregated metrics and per-case results are written out for later inspection.
9. Output and reporting
- The main evaluation artifacts are a CSV of run results and a scorecard/summary.
- scripts/visualize.py turns evaluator output into charts, so visualization is downstream of the benchmark and not part of the core execution loop.
10. CI integration
- .github/workflows/eval.yml wires the whole system into GitHub Actions.
- The CI flow runs Go tests, starts the mock Python API, executes the evaluation suite, and uploads artifacts.
- This makes the evaluator itself part of the project’s regression gate.
11. Tooling and runtime split
- The project intentionally uses two runtimes: Go for the orchestrator/metrics, Python for model serving and validation.
- That split is architecturally meaningful: Go handles deterministic batch evaluation, while Python hosts the model wrapper and file-format checks.
12. Operational modes
- Local development can run fully without GPU by using mock mode.
- Real-model mode swaps in an actual LLM endpoint, but the evaluator interface stays the same.
- This stable API boundary is what makes CI reproducible even though the production path may depend on expensive model inference.
13. What to diagram
- Show ground_truth.json feeding the Go evaluator.
- Show the evaluator calling the Python /generate API.
- Show the validator sitting inside the Python service path.
- Show outputs flowing from evaluator to CSV/scorecard, then to visualization and CI artifacts.
- Treat GitHub Actions as the orchestrating environment that ties the pieces together on every push.
Main entry points: main.go Evaluator · app.py Generate API · eval.yml GitHub Actions · Makefile Makefile
Components 15
Python service
Inference and output validation boundary.
- Generate API · HTTP endpoint that accepts prompts and returns hex-encoded WAV payloads.
python_app/app.py - Validator · Checks RIFF/WAVE header correctness before outputs are accepted.
python_app/validator.py - LLM backend · Real-model execution path used when mock mode is disabled.
- Mock mode · Deterministic local/CI mode for serving predictable responses.
Go evaluator
Benchmark orchestration, scoring, and result shaping.
- Evaluator · Loads cases, calls the Python service, scores results, and emits artifacts.
evaluator/main.go - Metrics · Computes similarity and latency statistics for benchmark runs.
evaluator/metrics.go - Data types · Shared structures for dataset entries, responses, and result rows.
evaluator/types.go - Config · Evaluator settings and run parameters.
evaluator/config.json - Metrics tests · Unit tests that protect scoring correctness.
evaluator/metrics_test.go
Benchmark data
Read-only corpus used for repeatable evaluation.
- Ground truth · Canonical benchmark corpus consumed by CI and local runs.
data/ground_truth.json
Automation
Local and CI entrypoints plus downstream reporting.
- Results CSV · Per-case benchmark output written by the evaluator.
- Scorecard · Aggregated benchmark summary for inspection and comparison.
- Visualizer · Converts evaluator output into charts and plots.
scripts/visualize.py - GitHub Actions · Automated regression flow that runs tests, evaluation, and artifact upload.
.github/workflows/eval.yml - Makefile · Convenience target wrapper for local execution.
Makefile
Connections 18
- Ground truth → Evaluator: benchmark cases · The evaluator reads the canonical corpus as input.
- Evaluator → Generate API: prompts · Each case is sent to the Python generation endpoint.
- Generate API → Validator: validate · Generated payloads pass through file-format checks.
- Generate API → LLM backend: real mode · Production path forwards to a live LLM backend.
- Generate API → Mock mode: mock mode · CI and local runs can use deterministic mocked responses.
- Evaluator → Metrics: score · Similarity and latency metrics are computed from responses.
- Evaluator → Data types: structures · Shared types shape requests, responses, and result rows.
- Evaluator → Results CSV: write · Per-case results are persisted for later inspection.
- Evaluator → Scorecard: summarize · Aggregate metrics are emitted as a run summary.
- Results CSV → Visualizer: analyze · Visualization consumes evaluator output artifacts.
- Scorecard → Visualizer: chart · Summary data can be turned into comparison graphics.
- GitHub Actions → Metrics tests: run tests · CI verifies scoring logic before evaluation.
- GitHub Actions → Mock mode: start mock · Workflow boots the mock Python service for reproducible runs.
- GitHub Actions → Evaluator: execute eval · GitHub Actions runs the benchmark orchestrator.
- GitHub Actions → Results CSV: upload artifact · Benchmark outputs are collected as CI artifacts.
- Makefile → Evaluator: local run · Make targets provide a convenient local execution path.
- Makefile → Generate API: serve app · Local development can start the Python service from the same entrypoint.
- Config → Evaluator: configure · Runtime settings tune the evaluator run.