janiduimesh/llm-eval-ci-cd-pipeline

How janiduimesh/llm-eval-ci-cd-pipeline works

This repository evaluates a university-rules RAG chatbot against a golden question set, then reports whether its answers meet quality and operational thresholds. The main path is the evaluation entry point through dataset loading, retrieval and answer generation, per-question scoring, aggregate threshold checks, and report persistence; a Streamlit dashboard presents stored results and trends. The rules corpus is indexed in persistent ChromaDB using Gemini embeddings, while Groq supplies generated answers and LLM-based judging. README-documented CI integration is included; unsampled workflow internals are not inferred.

Main entry points: run_eval.py Evaluation Runner · retriever.py Rules Retriever · rag_chatbot.py RAG Chatbot · report_generator.py Report Generator

Components 18

RAG Answering

RAG Chatbot src/rag_chatbot.py
Rules Retriever src/retriever.py
Academic Rules knowledge_base
Persistent ChromaDB src/retriever.py
Gemini Embeddings
Groq Answer Client src/llm_client.py
Groq API

Evaluation Engine

Evaluation Entry main.py
Evaluation Runner evaluation/run_eval.py
Golden Questions eval_data/golden_dataset.json
Answer Metrics evaluation/metrics.py
Threshold Checker evaluation/threshold_checker.py

Reporting and Dashboard

Report Generator evaluation/report_generator.py
Question Results results/eval_results.json
Run History results/history.csv
Trend Dashboard dashboard/app.py

CI Integration

GitHub Actions .github/workflows/llm_eval.yml

Other components

Evaluator

Connections 19

Evaluator → Evaluation Entry: starts evaluation README.md
Evaluation Entry → Evaluation Runner: calls run main.py
Evaluation Runner → Golden Questions: loads questions evaluation/run_eval.py
Evaluation Runner → RAG Chatbot: requests answers evaluation/run_eval.py
RAG Chatbot → Rules Retriever: retrieves context src/rag_chatbot.py
Rules Retriever → Academic Rules: indexes rules src/retriever.py
Rules Retriever → Persistent ChromaDB: indexes and queries src/retriever.py
Rules Retriever → Gemini Embeddings: embeds text src/retriever.py
RAG Chatbot → Groq Answer Client: generates answer src/rag_chatbot.py
Groq Answer Client → Groq API: sends prompt src/llm_client.py
Evaluation Runner → Answer Metrics: scores answers evaluation/run_eval.py
Answer Metrics → Groq API: requests judge scores evaluation/metrics.py
Evaluation Runner → Threshold Checker: checks aggregate evaluation/run_eval.py
Evaluation Runner → Report Generator: emits reports evaluation/run_eval.py
Report Generator → Question Results: writes results evaluation/report_generator.py
Report Generator → Run History: appends summary evaluation/report_generator.py
Trend Dashboard → Question Results: reads results README.md
Trend Dashboard → Run History: reads trends README.md
GitHub Actions → Evaluation Entry: runs evaluation README.md

Drawn by GitDiagram on November 15, 2026 from the repository's file tree, README and source files. Read as Markdown ·