How dkbattulas/artificial-intelligence-eval-harness works
The product is a FastAPI-based LLM evaluation harness. Its principal workflow loads the configured JSON test set, generates answers through Groq, evaluates them against ground truth, and returns structured scores. The README also describes dataset input and dashboard output. The sampled source confirms the evaluation and serving path; the dashboard’s client-side wiring and several README-described metrics are not visible, so those connections are omitted.
Main entry points: evaluate.py Evaluation endpoint · evaluation_service.py Evaluation pipeline · main.py FastAPI app · llm_service.py Answer generation
Components 14
Serving
- FastAPI app
app/main.py - Dashboard
static/index.html
Evaluation workflow
- Evaluation endpoint
app/routes/evaluate.py - JSON test set
tests/sample_testset.json - Answer generation
app/services/llm_service.py - Structured results
app/routes/evaluate.py
Scoring
- Evaluation pipeline
app/services/evaluation_service.py - Keyword metrics
app/services/evaluation_service.py - Embedding similarity
app/services/evaluation_service.py - Local judge
app/services/llm_judge.py - Score aggregation
app/services/evaluation_service.py
External models
- Groq API
- Sentence model
Other components
- User
Connections 13
- User → FastAPI app: requests
- FastAPI app → Dashboard: serves
- User → Evaluation endpoint: runs evaluation
- Evaluation endpoint → JSON test set: reads
- Evaluation endpoint → Answer generation: requests answers
- Answer generation → Groq API: generates answer
- Evaluation endpoint → Evaluation pipeline: evaluates responses
- Evaluation pipeline → Keyword metrics: extracts terms
- Evaluation pipeline → Embedding similarity: encodes text
- Embedding similarity → Sentence model: encodes
- Evaluation pipeline → Local judge: requests judgment
- Evaluation pipeline → Score aggregation: combines scores
- Evaluation endpoint → Structured results: returns JSON