# vkr-vavilla/llm_evaluation architecture

> Architecture diagram of the GitHub repository vkr-vavilla/llm_evaluation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vkr-vavilla/llm_evaluation
- Repository: https://github.com/vkr-vavilla/llm_evaluation

## Overview

This repository is a two-part evaluation pipeline: a small Python inference service and a Go-based evaluator that benchmarks that service against a fixed ground-truth corpus. The overall architecture is designed for repeatable QA of LLM-generated WAV exploit payloads.

1. **Top-level split: service vs. evaluator**
   - `python_app/` is the model-facing API layer.
   - `evaluator/` is the benchmarking and reporting engine.
   - `data/ground_truth.json` is the stable input dataset both CI and local runs consume.
   - `scripts/visualize.py` is a post-processing/reporting tool, not part of the evaluation path itself.

2. **Python inference boundary**
   - `python_app/app.py` exposes a `/generate` endpoint.
   - It can run in **mock mode** for CI/local testing or with a real LLM backend such as CodeLlama.
   - This is the main external boundary the evaluator talks to: it receives prompts and returns hex-encoded WAV payloads.

3. **Payload validation layer**
   - `python_app/validator.py` enforces RIFF/WAVE header correctness on generated outputs.
   - This creates an early correctness gate before results are scored, so the system tracks both semantic similarity and basic file-structure validity.

4. **Go evaluation engine**
   - `evaluator/main.go` orchestrates the benchmark run: it loads the dataset, sends prompts to the Python API, gathers responses, computes metrics, and emits results.
   - The evaluator is the control plane for the whole workflow and likely the component that writes `results.csv` and the scorecard mentioned in the README.

5. **Metric computation subsystem**
   - `evaluator/metrics.go` contains the scoring logic: normalized Levenshtein similarity, latency percentiles, and standard deviation calculations.
   - `evaluator/metrics_test.go` provides unit coverage for these computations, which is important because the evaluation output is only as trustworthy as the metric code.

6. **Data model boundary**
   - `evaluator/types.go` defines the shared structures used by the evaluator for dataset entries, responses, and result rows.
   - This keeps the run logic decoupled from the dataset format and output format, which matters for diagramming the data flow cleanly.

7. **Dataset role**
   - `data/ground_truth.json` is the canonical benchmark corpus: crash-inducing payloads plus valid/edge-case WAVs.
   - The evaluator treats this as read-only input, so the repository has a strong separation between immutable benchmark data and generated outputs.

8. **Execution flow**
   - A run starts with the evaluator reading `ground_truth.json`.
   - For each case, it sends a prompt to the Python `/generate` endpoint.
   - The response is validated and scored against ground truth.
   - Aggregated metrics and per-case results are written out for later inspection.

9. **Output and reporting**
   - The main evaluation artifacts are a CSV of run results and a scorecard/summary.
   - `scripts/visualize.py` turns evaluator output into charts, so visualization is downstream of the benchmark and not part of the core execution loop.

10. **CI integration**
   - `.github/workflows/eval.yml` wires the whole system into GitHub Actions.
   - The CI flow runs Go tests, starts the mock Python API, executes the evaluation suite, and uploads artifacts.
   - This makes the evaluator itself part of the project’s regression gate.

11. **Tooling and runtime split**
   - The project intentionally uses two runtimes: Go for the orchestrator/metrics, Python for model serving and validation.
   - That split is architecturally meaningful: Go handles deterministic batch evaluation, while Python hosts the model wrapper and file-format checks.

12. **Operational modes**
   - Local development can run fully without GPU by using mock mode.
   - Real-model mode swaps in an actual LLM endpoint, but the evaluator interface stays the same.
   - This stable API boundary is what makes CI reproducible even though the production path may depend on expensive model inference.

13. **What to diagram**
   - Show `ground_truth.json` feeding the Go evaluator.
   - Show the evaluator calling the Python `/generate` API.
   - Show the validator sitting inside the Python service path.
   - Show outputs flowing from evaluator to CSV/scorecard, then to visualization and CI artifacts.
   - Treat GitHub Actions as the orchestrating environment that ties the pieces together on every push.

Main entry points: [`evaluator/main.go`](https://github.com/vkr-vavilla/llm_evaluation/blob/main/evaluator/main.go) (Evaluator), [`python_app/app.py`](https://github.com/vkr-vavilla/llm_evaluation/blob/main/python_app/app.py) (Generate API), [`.github/workflows/eval.yml`](https://github.com/vkr-vavilla/llm_evaluation/blob/main/.github/workflows/eval.yml) (GitHub Actions), [`Makefile`](https://github.com/vkr-vavilla/llm_evaluation/tree/main/Makefile) (Makefile)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_python["Python service"]
  node_node_python_api["Generate API<br/>python service<br/>[app.py]"]
  node_node_python_validator["Validator<br/>WAV guard<br/>[validator.py]"]
  node_node_python_backend(("LLM backend<br/>model mode"))
  node_node_python_mock(("Mock mode<br/>test mode"))
end

subgraph group_group_evaluator["Go evaluator"]
  node_node_eval_main["Evaluator<br/>run orchestrator<br/>[main.go]"]
  node_node_eval_metrics["Metrics<br/>scoring<br/>[metrics.go]"]
  node_node_eval_types["Data types<br/>schema<br/>[types.go]"]
  node_node_eval_config["Config<br/>[config.json]"]
  node_node_go_test["Metrics tests<br/>verification<br/>[metrics_test.go]"]
end

subgraph group_group_data["Benchmark data"]
  node_node_dataset[("Ground truth<br/>corpus<br/>[ground_truth.json]")]
end

subgraph group_group_ops["Automation"]
  node_node_results_csv["Results CSV<br/>run artifact"]
  node_node_scorecard["Scorecard<br/>summary artifact"]
  node_node_visualize["Visualizer<br/>reporting<br/>[visualize.py]"]
  node_node_ci["GitHub Actions<br/>ci workflow<br/>[eval.yml]"]
  node_node_makefile["Makefile<br/>local entrypoint"]
end

node_node_dataset -->|"benchmark cases"| node_node_eval_main
node_node_eval_main -->|"prompts"| node_node_python_api
node_node_python_api -->|"validate"| node_node_python_validator
node_node_python_api -->|"real mode"| node_node_python_backend
node_node_python_api -->|"mock mode"| node_node_python_mock
node_node_eval_main -->|"score"| node_node_eval_metrics
node_node_eval_main -->|"structures"| node_node_eval_types
node_node_eval_main -->|"write"| node_node_results_csv
node_node_eval_main -->|"summarize"| node_node_scorecard
node_node_results_csv -->|"analyze"| node_node_visualize
node_node_scorecard -->|"chart"| node_node_visualize
node_node_ci -->|"run tests"| node_node_go_test
node_node_ci -->|"start mock"| node_node_python_mock
node_node_ci -->|"execute eval"| node_node_eval_main
node_node_ci -->|"upload artifact"| node_node_results_csv
node_node_makefile -->|"local run"| node_node_eval_main
node_node_makefile -->|"serve app"| node_node_python_api
node_node_eval_config -->|"configure"| node_node_eval_main

click node_node_python_api "https://github.com/vkr-vavilla/llm_evaluation/blob/main/python_app/app.py"
click node_node_python_validator "https://github.com/vkr-vavilla/llm_evaluation/blob/main/python_app/validator.py"
click node_node_eval_main "https://github.com/vkr-vavilla/llm_evaluation/blob/main/evaluator/main.go"
click node_node_eval_metrics "https://github.com/vkr-vavilla/llm_evaluation/blob/main/evaluator/metrics.go"
click node_node_eval_types "https://github.com/vkr-vavilla/llm_evaluation/blob/main/evaluator/types.go"
click node_node_eval_config "https://github.com/vkr-vavilla/llm_evaluation/blob/main/evaluator/config.json"
click node_node_dataset "https://github.com/vkr-vavilla/llm_evaluation/blob/main/data/ground_truth.json"
click node_node_visualize "https://github.com/vkr-vavilla/llm_evaluation/blob/main/scripts/visualize.py"
click node_node_ci "https://github.com/vkr-vavilla/llm_evaluation/blob/main/.github/workflows/eval.yml"
click node_node_makefile "https://github.com/vkr-vavilla/llm_evaluation/tree/main/Makefile"
click node_node_go_test "https://github.com/vkr-vavilla/llm_evaluation/blob/main/evaluator/metrics_test.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_python_api,node_node_python_validator,node_node_python_backend,node_node_python_mock toneBlue
class node_node_eval_main,node_node_eval_metrics,node_node_eval_types,node_node_eval_config,node_node_go_test toneAmber
class node_node_dataset toneMint
class node_node_results_csv,node_node_scorecard,node_node_visualize,node_node_ci,node_node_makefile toneRose
```

## Components

### Python service

Inference and output validation boundary.

- [Generate API](https://github.com/vkr-vavilla/llm_evaluation/blob/main/python_app/app.py): python service · HTTP endpoint that accepts prompts and returns hex-encoded WAV payloads. · `python_app/app.py`
- [Validator](https://github.com/vkr-vavilla/llm_evaluation/blob/main/python_app/validator.py): WAV guard · Checks RIFF/WAVE header correctness before outputs are accepted. · `python_app/validator.py`
- **LLM backend**: model mode · Real-model execution path used when mock mode is disabled.
- **Mock mode**: test mode · Deterministic local/CI mode for serving predictable responses.

### Go evaluator

Benchmark orchestration, scoring, and result shaping.

- [Evaluator](https://github.com/vkr-vavilla/llm_evaluation/blob/main/evaluator/main.go): run orchestrator · Loads cases, calls the Python service, scores results, and emits artifacts. · `evaluator/main.go`
- [Metrics](https://github.com/vkr-vavilla/llm_evaluation/blob/main/evaluator/metrics.go): scoring · Computes similarity and latency statistics for benchmark runs. · `evaluator/metrics.go`
- [Data types](https://github.com/vkr-vavilla/llm_evaluation/blob/main/evaluator/types.go): schema · Shared structures for dataset entries, responses, and result rows. · `evaluator/types.go`
- [Config](https://github.com/vkr-vavilla/llm_evaluation/blob/main/evaluator/config.json): runtime config · Evaluator settings and run parameters. · `evaluator/config.json`
- [Metrics tests](https://github.com/vkr-vavilla/llm_evaluation/blob/main/evaluator/metrics_test.go): verification · Unit tests that protect scoring correctness. · `evaluator/metrics_test.go`

### Benchmark data

Read-only corpus used for repeatable evaluation.

- [Ground truth](https://github.com/vkr-vavilla/llm_evaluation/blob/main/data/ground_truth.json): corpus · Canonical benchmark corpus consumed by CI and local runs. · `data/ground_truth.json`

### Automation

Local and CI entrypoints plus downstream reporting.

- **Results CSV**: run artifact · Per-case benchmark output written by the evaluator.
- **Scorecard**: summary artifact · Aggregated benchmark summary for inspection and comparison.
- [Visualizer](https://github.com/vkr-vavilla/llm_evaluation/blob/main/scripts/visualize.py): reporting · Converts evaluator output into charts and plots. · `scripts/visualize.py`
- [GitHub Actions](https://github.com/vkr-vavilla/llm_evaluation/blob/main/.github/workflows/eval.yml): ci workflow · Automated regression flow that runs tests, evaluation, and artifact upload. · `.github/workflows/eval.yml`
- [Makefile](https://github.com/vkr-vavilla/llm_evaluation/tree/main/Makefile): local entrypoint · Convenience target wrapper for local execution. · `Makefile`

## Connections

- Ground truth → Evaluator: benchmark cases · The evaluator reads the canonical corpus as input.
- Evaluator → Generate API: prompts · Each case is sent to the Python generation endpoint.
- Generate API → Validator: validate · Generated payloads pass through file-format checks.
- Generate API → LLM backend: real mode · Production path forwards to a live LLM backend.
- Generate API → Mock mode: mock mode · CI and local runs can use deterministic mocked responses.
- Evaluator → Metrics: score · Similarity and latency metrics are computed from responses.
- Evaluator → Data types: structures · Shared types shape requests, responses, and result rows.
- Evaluator → Results CSV: write · Per-case results are persisted for later inspection.
- Evaluator → Scorecard: summarize · Aggregate metrics are emitted as a run summary.
- Results CSV → Visualizer: analyze · Visualization consumes evaluator output artifacts.
- Scorecard → Visualizer: chart · Summary data can be turned into comparison graphics.
- GitHub Actions → Metrics tests: run tests · CI verifies scoring logic before evaluation.
- GitHub Actions → Mock mode: start mock · Workflow boots the mock Python service for reproducible runs.
- GitHub Actions → Evaluator: execute eval · GitHub Actions runs the benchmark orchestrator.
- GitHub Actions → Results CSV: upload artifact · Benchmark outputs are collected as CI artifacts.
- Makefile → Evaluator: local run · Make targets provide a convenient local execution path.
- Makefile → Generate API: serve app · Local development can start the Python service from the same entrypoint.
- Config → Evaluator: configure · Runtime settings tune the evaluator run.
