# janiduimesh/llm-eval-ci-cd-pipeline architecture

> Architecture diagram of the GitHub repository janiduimesh/llm-eval-ci-cd-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/janiduimesh/llm-eval-ci-cd-pipeline
- Repository: https://github.com/janiduimesh/llm-eval-ci-cd-pipeline

## Overview

This repository evaluates a university-rules RAG chatbot against a golden question set, then reports whether its answers meet quality and operational thresholds. The main path is the evaluation entry point through dataset loading, retrieval and answer generation, per-question scoring, aggregate threshold checks, and report persistence; a Streamlit dashboard presents stored results and trends. The rules corpus is indexed in persistent ChromaDB using Gemini embeddings, while Groq supplies generated answers and LLM-based judging. README-documented CI integration is included; unsampled workflow internals are not inferred.

Main entry points: [`evaluation/run_eval.py`](https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/evaluation/run_eval.py) (Evaluation Runner), [`src/retriever.py`](https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/src/retriever.py) (Rules Retriever), [`src/rag_chatbot.py`](https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/src/rag_chatbot.py) (RAG Chatbot), [`evaluation/report_generator.py`](https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/evaluation/report_generator.py) (Report Generator)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_rag["RAG Answering"]
  node_chatbot["RAG Chatbot<br/>[rag_chatbot.py]"]
  node_retriever["Rules Retriever<br/>[retriever.py]"]
  node_rules["Academic Rules"]
  node_chroma[("Persistent ChromaDB<br/>[retriever.py]")]
  node_gemini{{"Gemini Embeddings"}}
  node_llm["Groq Answer Client<br/>[llm_client.py]"]
  node_groq{{"Groq API"}}
end

subgraph group_evaluation["Evaluation Engine"]
  node_entry["Evaluation Entry<br/>[main.py]"]
  node_runner["Evaluation Runner<br/>[run_eval.py]"]
  node_dataset["Golden Questions"]
  node_metrics["Answer Metrics<br/>[metrics.py]"]
  node_thresholds["Threshold Checker"]
end

subgraph group_reporting["Reporting and Dashboard"]
  node_reports["Report Generator"]
  node_jsonresults["Question Results<br/>[eval_results.json]"]
  node_history["Run History<br/>[history.csv]"]
  node_dashboard["Trend Dashboard<br/>[app.py]"]
end

subgraph group_automation["CI Integration"]
  node_actions["GitHub Actions<br/>[llm_eval.yml]"]
end

node_user(("Evaluator"))

node_user -->|"starts evaluation"| node_entry
node_entry -->|"calls run"| node_runner
node_runner -->|"loads questions"| node_dataset
node_runner -->|"requests answers"| node_chatbot
node_chatbot -->|"retrieves context"| node_retriever
node_retriever -->|"indexes rules"| node_rules
node_retriever -->|"indexes and queries"| node_chroma
node_retriever -->|"embeds text"| node_gemini
node_chatbot -->|"generates answer"| node_llm
node_llm -->|"sends prompt"| node_groq
node_runner -->|"scores answers"| node_metrics
node_metrics -->|"requests judge scores"| node_groq
node_runner -->|"checks aggregate"| node_thresholds
node_runner -->|"emits reports"| node_reports
node_reports -->|"writes results"| node_jsonresults
node_reports -->|"appends summary"| node_history
node_dashboard -->|"reads results"| node_jsonresults
node_dashboard -->|"reads trends"| node_history
node_actions -.->|"runs evaluation"| node_entry

click node_entry "https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/main.py"
click node_runner "https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/evaluation/run_eval.py"
click node_dataset "https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/eval_data/golden_dataset.json"
click node_chatbot "https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/src/rag_chatbot.py"
click node_retriever "https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/src/retriever.py"
click node_rules "https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/tree/main/knowledge_base"
click node_chroma "https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/src/retriever.py"
click node_llm "https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/src/llm_client.py"
click node_metrics "https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/evaluation/metrics.py"
click node_thresholds "https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/evaluation/threshold_checker.py"
click node_reports "https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/evaluation/report_generator.py"
click node_jsonresults "https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/results/eval_results.json"
click node_history "https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/results/history.csv"
click node_dashboard "https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/dashboard/app.py"
click node_actions "https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/.github/workflows/llm_eval.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_chatbot,node_retriever,node_rules,node_chroma,node_gemini,node_llm,node_groq toneBlue
class node_entry,node_runner,node_dataset,node_metrics,node_thresholds toneAmber
class node_reports,node_jsonresults,node_history,node_dashboard toneMint
class node_actions toneRose
class node_user toneIndigo
```

## Components

### RAG Answering

- [RAG Chatbot](https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/src/rag_chatbot.py): `src/rag_chatbot.py`
- [Rules Retriever](https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/src/retriever.py): `src/retriever.py`
- [Academic Rules](https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/tree/main/knowledge_base): `knowledge_base`
- [Persistent ChromaDB](https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/src/retriever.py): `src/retriever.py`
- **Gemini Embeddings**
- [Groq Answer Client](https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/src/llm_client.py): `src/llm_client.py`
- **Groq API**

### Evaluation Engine

- [Evaluation Entry](https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/main.py): `main.py`
- [Evaluation Runner](https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/evaluation/run_eval.py): `evaluation/run_eval.py`
- [Golden Questions](https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/eval_data/golden_dataset.json): `eval_data/golden_dataset.json`
- [Answer Metrics](https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/evaluation/metrics.py): `evaluation/metrics.py`
- [Threshold Checker](https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/evaluation/threshold_checker.py): `evaluation/threshold_checker.py`

### Reporting and Dashboard

- [Report Generator](https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/evaluation/report_generator.py): `evaluation/report_generator.py`
- [Question Results](https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/results/eval_results.json): `results/eval_results.json`
- [Run History](https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/results/history.csv): `results/history.csv`
- [Trend Dashboard](https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/dashboard/app.py): `dashboard/app.py`

### CI Integration

- [GitHub Actions](https://github.com/janiduimesh/llm-eval-ci-cd-pipeline/blob/main/.github/workflows/llm_eval.yml): `.github/workflows/llm_eval.yml`

### Other components

- **Evaluator**

## Connections

- Evaluator → Evaluation Entry: starts evaluation (evidence: `README.md`)
- Evaluation Entry → Evaluation Runner: calls run (evidence: `main.py`)
- Evaluation Runner → Golden Questions: loads questions (evidence: `evaluation/run_eval.py`)
- Evaluation Runner → RAG Chatbot: requests answers (evidence: `evaluation/run_eval.py`)
- RAG Chatbot → Rules Retriever: retrieves context (evidence: `src/rag_chatbot.py`)
- Rules Retriever → Academic Rules: indexes rules (evidence: `src/retriever.py`)
- Rules Retriever → Persistent ChromaDB: indexes and queries (evidence: `src/retriever.py`)
- Rules Retriever → Gemini Embeddings: embeds text (evidence: `src/retriever.py`)
- RAG Chatbot → Groq Answer Client: generates answer (evidence: `src/rag_chatbot.py`)
- Groq Answer Client → Groq API: sends prompt (evidence: `src/llm_client.py`)
- Evaluation Runner → Answer Metrics: scores answers (evidence: `evaluation/run_eval.py`)
- Answer Metrics → Groq API: requests judge scores (evidence: `evaluation/metrics.py`)
- Evaluation Runner → Threshold Checker: checks aggregate (evidence: `evaluation/run_eval.py`)
- Evaluation Runner → Report Generator: emits reports (evidence: `evaluation/run_eval.py`)
- Report Generator → Question Results: writes results (evidence: `evaluation/report_generator.py`)
- Report Generator → Run History: appends summary (evidence: `evaluation/report_generator.py`)
- Trend Dashboard → Question Results: reads results (evidence: `README.md`)
- Trend Dashboard → Run History: reads trends (evidence: `README.md`)
- GitHub Actions → Evaluation Entry: runs evaluation (evidence: `README.md`)