# dkbattulas/artificial-intelligence-eval-harness architecture

> Architecture diagram of the GitHub repository dkbattulas/artificial-intelligence-eval-harness, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dkbattulas/artificial-intelligence-eval-harness
- Repository: https://github.com/dkbattulas/artificial-intelligence-eval-harness

## Overview

The product is a FastAPI-based LLM evaluation harness. Its principal workflow loads the configured JSON test set, generates answers through Groq, evaluates them against ground truth, and returns structured scores. The README also describes dataset input and dashboard output. The sampled source confirms the evaluation and serving path; the dashboard’s client-side wiring and several README-described metrics are not visible, so those connections are omitted.

Main entry points: [`app/routes/evaluate.py`](https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/app/routes/evaluate.py) (Evaluation endpoint), [`app/services/evaluation_service.py`](https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/app/services/evaluation_service.py) (Evaluation pipeline), [`app/main.py`](https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/app/main.py) (FastAPI app), [`app/services/llm_service.py`](https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/app/services/llm_service.py) (Answer generation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_serving["Serving"]
  node_app["FastAPI app<br/>[main.py]"]
  node_dashboard["Dashboard<br/>[index.html]"]
end

subgraph group_workflow["Evaluation workflow"]
  node_endpoint["Evaluation endpoint<br/>[evaluate.py]"]
  node_dataset["JSON test set"]
  node_generation["Answer generation<br/>[llm_service.py]"]
  node_results["Structured results<br/>[evaluate.py]"]
end

subgraph group_scoring["Scoring"]
  node_evaluation["Evaluation pipeline"]
  node_keywords["Keyword metrics"]
  node_embeddings["Embedding similarity"]
  node_judge["Local judge<br/>[llm_judge.py]"]
  node_aggregation["Score aggregation"]
end

subgraph group_integrations["External models"]
  node_groq{{"Groq API"}}
  node_sentence_model{{"Sentence model"}}
end

node_user(("User"))

node_user -->|"requests"| node_app
node_app -->|"serves"| node_dashboard
node_user -->|"runs evaluation"| node_endpoint
node_endpoint -->|"reads"| node_dataset
node_endpoint -->|"requests answers"| node_generation
node_generation -->|"generates answer"| node_groq
node_endpoint -->|"evaluates responses"| node_evaluation
node_evaluation -->|"extracts terms"| node_keywords
node_evaluation -->|"encodes text"| node_embeddings
node_embeddings -->|"encodes"| node_sentence_model
node_evaluation -->|"requests judgment"| node_judge
node_evaluation -->|"combines scores"| node_aggregation
node_endpoint -->|"returns JSON"| node_results

click node_app "https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/app/main.py"
click node_dashboard "https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/static/index.html"
click node_endpoint "https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/app/routes/evaluate.py"
click node_dataset "https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/tests/sample_testset.json"
click node_generation "https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/app/services/llm_service.py"
click node_evaluation "https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/app/services/evaluation_service.py"
click node_keywords "https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/app/services/evaluation_service.py"
click node_embeddings "https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/app/services/evaluation_service.py"
click node_judge "https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/app/services/llm_judge.py"
click node_aggregation "https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/app/services/evaluation_service.py"
click node_results "https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/app/routes/evaluate.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app,node_dashboard,node_user toneBlue
class node_endpoint,node_dataset,node_generation,node_results toneAmber
class node_evaluation,node_keywords,node_embeddings,node_judge,node_aggregation toneMint
class node_groq,node_sentence_model toneRose
```

## Components

### Serving

- [FastAPI app](https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/app/main.py): `app/main.py`
- [Dashboard](https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/static/index.html): `static/index.html`

### Evaluation workflow

- [Evaluation endpoint](https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/app/routes/evaluate.py): `app/routes/evaluate.py`
- [JSON test set](https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/tests/sample_testset.json): `tests/sample_testset.json`
- [Answer generation](https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/app/services/llm_service.py): `app/services/llm_service.py`
- [Structured results](https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/app/routes/evaluate.py): `app/routes/evaluate.py`

### Scoring

- [Evaluation pipeline](https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/app/services/evaluation_service.py): `app/services/evaluation_service.py`
- [Keyword metrics](https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/app/services/evaluation_service.py): `app/services/evaluation_service.py`
- [Embedding similarity](https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/app/services/evaluation_service.py): `app/services/evaluation_service.py`
- [Local judge](https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/app/services/llm_judge.py): `app/services/llm_judge.py`
- [Score aggregation](https://github.com/dkbattulas/artificial-intelligence-eval-harness/blob/main/app/services/evaluation_service.py): `app/services/evaluation_service.py`

### External models

- **Groq API**
- **Sentence model**

### Other components

- **User**

## Connections

- User → FastAPI app: requests
- FastAPI app → Dashboard: serves
- User → Evaluation endpoint: runs evaluation
- Evaluation endpoint → JSON test set: reads
- Evaluation endpoint → Answer generation: requests answers
- Answer generation → Groq API: generates answer
- Evaluation endpoint → Evaluation pipeline: evaluates responses
- Evaluation pipeline → Keyword metrics: extracts terms
- Evaluation pipeline → Embedding similarity: encodes text
- Embedding similarity → Sentence model: encodes
- Evaluation pipeline → Local judge: requests judgment
- Evaluation pipeline → Score aggregation: combines scores
- Evaluation endpoint → Structured results: returns JSON