# 2501pr0ject/ragnarok-ai architecture

> Architecture diagram of the GitHub repository 2501pr0ject/ragnarok-ai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/2501pr0ject/ragnarok-ai
- Repository: https://github.com/2501pr0ject/ragnarok-ai

## Overview

RAGnarok-AI is a local-first framework for evaluating RAG systems. The main workflow is to provide a test set and RAG pipeline, run retrieval and generation-quality evaluation (optionally with a local LLM judge), then inspect or export results. Separate capabilities generate or mine test sets, calibrate judges, diagnose failures, and compare or monitor configurations for drift and production health. The graph combines sampled call evidence with README-described capabilities; several feature implementations were not sampled, so their internal connections are omitted.

Main entry points: [`src/ragnarok_ai/core/evaluate.py`](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/core/evaluate.py) (Evaluation runner), [`src/ragnarok_ai/monitor/models.py`](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/monitor/models.py) (Trace models), [`src/ragnarok_ai/evaluators/judge.py`](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/evaluators/judge.py) (Generation evaluator), [`src/ragnarok_ai/core/types.py`](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/core/types.py) (Test set types)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_evaluation["Evaluation workflow"]
  node_testset["Test set types<br/>[types.py]"]
  node_evaluate["Evaluation runner<br/>[evaluate.py]"]
  node_retrieval["Retrieval metrics<br/>[retrieval.py]"]
  node_cost["Cost tracking<br/>[tracker.py]"]
  node_provenance["Evaluation provenance<br/>[provenance.py]"]
  node_exceptions["Error types<br/>[exceptions.py]"]
end

subgraph group_judging["Judging and trust"]
  node_judgeeval["Generation evaluator<br/>[judge.py]"]
  node_judgecontract["Judge contracts<br/>[contract.py]"]
  node_hhem["HHEM judge<br/>[hhem.py]"]
  node_minicheck["MiniCheck judge<br/>[minicheck.py]"]
  node_segmentation["Text segmentation<br/>[segmentation.py]"]
  node_calibration["Judge calibration<br/>[calibrator.py]"]
  node_diagnosis["Root-cause diagnosis<br/>[diagnostician.py]"]
end

subgraph group_testdata["Test data"]
  node_generator["Test generation"]
  node_miner["Traffic mining<br/>[miner.py]"]
  node_datasetio["Dataset I/O<br/>[io.py]"]
end

subgraph group_operations["Production intelligence"]
  node_monitorclient["Trace collection<br/>[client.py]"]
  node_monitorstore[("Trace store<br/>[store.py]")]
  node_monitormodels["Trace models<br/>[models.py]"]
  node_drift["Drift detection<br/>[detector.py]"]
  node_ab["A/B analysis<br/>[analyzer.py]"]
  node_alerts["Alert management<br/>[manager.py]"]
  node_privacy["Privacy controls<br/>[privacy.py]"]
  node_abmodels["Experiment models<br/>[models.py]"]
end

subgraph group_interfaces["Interfaces and integrations"]
  node_cli["CLI commands<br/>[main.py]"]
  node_reporters["Result reporters<br/>[json.py]"]
  node_console["Console reporter<br/>[console.py]"]
  node_adapters["Framework adapters"]
  node_llmadapters["LLM adapters"]
  node_protocols["Extension protocols<br/>[protocols.py]"]
end

node_user(("Engineer"))
node_pipeline["RAG pipeline"]

node_user -->|"runs"| node_cli
node_user -->|"invokes"| node_evaluate
node_cli -->|"dispatches"| node_evaluate
node_testset -->|"supplies queries"| node_evaluate
node_evaluate -->|"queries"| node_pipeline
node_evaluate -->|"computes metrics"| node_retrieval
node_evaluate -.->|"scores generation"| node_judgeeval
node_evaluate -->|"tracks usage"| node_cost
node_evaluate -->|"returns results"| node_reporters
node_reporters -->|"uses"| node_console
node_judgeeval -->|"uses contract"| node_judgecontract
node_hhem -->|"implements"| node_judgecontract
node_hhem -->|"segments text"| node_segmentation
node_minicheck -->|"uses contract"| node_judgecontract
node_judgeeval -->|"uses LLM client"| node_llmadapters
node_calibration -->|"evaluates samples"| node_judgeeval
node_diagnosis -->|"re-evaluates failures"| node_evaluate
node_generator -.->|"produces test sets"| node_testset
node_miner -->|"mines traces"| node_monitormodels
node_datasetio -->|"loads datasets"| node_testset
node_monitorclient -->|"creates traces"| node_monitormodels
node_monitorclient -->|"sanitizes data"| node_privacy
node_monitorstore -->|"stores records"| node_monitormodels
node_drift -->|"checks baseline"| node_monitormodels
node_drift -->|"raises alerts"| node_alerts
node_ab -->|"builds reports"| node_abmodels
node_ab -->|"compares runs"| node_monitormodels
node_pipeline -.->|"integrates through"| node_adapters
node_evaluate -.->|"records context"| node_provenance
node_evaluate -.->|"raises errors"| node_exceptions
node_adapters -.->|"implements interfaces"| node_protocols

click node_cli "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/cli/main.py"
click node_testset "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/core/types.py"
click node_evaluate "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/core/evaluate.py"
click node_retrieval "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/evaluators/retrieval.py"
click node_judgeeval "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/evaluators/judge.py"
click node_judgecontract "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/judges/contract.py"
click node_hhem "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/judges/hhem.py"
click node_minicheck "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/judges/minicheck.py"
click node_segmentation "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/text/segmentation.py"
click node_calibration "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/calibration/calibrator.py"
click node_diagnosis "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/diagnosis/diagnostician.py"
click node_reporters "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/reporters/json.py"
click node_console "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/reporters/console.py"
click node_generator "https://github.com/2501pr0ject/ragnarok-ai/tree/main/src/ragnarok_ai/generators"
click node_miner "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/mining/miner.py"
click node_datasetio "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/dataset/io.py"
click node_monitorclient "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/monitor/client.py"
click node_monitorstore "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/monitor/store.py"
click node_monitormodels "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/monitor/models.py"
click node_drift "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/drift/detector.py"
click node_ab "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/ab/analyzer.py"
click node_alerts "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/alerts/manager.py"
click node_privacy "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/privacy.py"
click node_cost "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/cost/tracker.py"
click node_adapters "https://github.com/2501pr0ject/ragnarok-ai/tree/main/src/ragnarok_ai/adapters/frameworks"
click node_llmadapters "https://github.com/2501pr0ject/ragnarok-ai/tree/main/src/ragnarok_ai/adapters/llm"
click node_provenance "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/core/provenance.py"
click node_exceptions "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/core/exceptions.py"
click node_protocols "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/core/protocols.py"
click node_abmodels "https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/ab/models.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_testset,node_evaluate,node_retrieval,node_cost,node_provenance,node_exceptions toneBlue
class node_judgeeval,node_judgecontract,node_hhem,node_minicheck,node_segmentation,node_calibration,node_diagnosis toneAmber
class node_generator,node_miner,node_datasetio toneMint
class node_monitorclient,node_monitorstore,node_monitormodels,node_drift,node_ab,node_alerts,node_privacy,node_abmodels toneRose
class node_cli,node_reporters,node_console,node_adapters,node_llmadapters,node_protocols,node_user,node_pipeline toneIndigo
```

## Components

### Evaluation workflow

- [Test set types](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/core/types.py): `src/ragnarok_ai/core/types.py`
- [Evaluation runner](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/core/evaluate.py): `src/ragnarok_ai/core/evaluate.py`
- [Retrieval metrics](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/evaluators/retrieval.py): `src/ragnarok_ai/evaluators/retrieval.py`
- [Cost tracking](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/cost/tracker.py): `src/ragnarok_ai/cost/tracker.py`
- [Evaluation provenance](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/core/provenance.py): `src/ragnarok_ai/core/provenance.py`
- [Error types](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/core/exceptions.py): `src/ragnarok_ai/core/exceptions.py`

### Judging and trust

- [Generation evaluator](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/evaluators/judge.py): `src/ragnarok_ai/evaluators/judge.py`
- [Judge contracts](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/judges/contract.py): `src/ragnarok_ai/judges/contract.py`
- [HHEM judge](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/judges/hhem.py): `src/ragnarok_ai/judges/hhem.py`
- [MiniCheck judge](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/judges/minicheck.py): `src/ragnarok_ai/judges/minicheck.py`
- [Text segmentation](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/text/segmentation.py): `src/ragnarok_ai/text/segmentation.py`
- [Judge calibration](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/calibration/calibrator.py): `src/ragnarok_ai/calibration/calibrator.py`
- [Root-cause diagnosis](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/diagnosis/diagnostician.py): `src/ragnarok_ai/diagnosis/diagnostician.py`

### Test data

- [Test generation](https://github.com/2501pr0ject/ragnarok-ai/tree/main/src/ragnarok_ai/generators): `src/ragnarok_ai/generators`
- [Traffic mining](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/mining/miner.py): `src/ragnarok_ai/mining/miner.py`
- [Dataset I/O](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/dataset/io.py): `src/ragnarok_ai/dataset/io.py`

### Production intelligence

- [Trace collection](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/monitor/client.py): `src/ragnarok_ai/monitor/client.py`
- [Trace store](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/monitor/store.py): `src/ragnarok_ai/monitor/store.py`
- [Trace models](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/monitor/models.py): `src/ragnarok_ai/monitor/models.py`
- [Drift detection](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/drift/detector.py): `src/ragnarok_ai/drift/detector.py`
- [A/B analysis](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/ab/analyzer.py): `src/ragnarok_ai/ab/analyzer.py`
- [Alert management](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/alerts/manager.py): `src/ragnarok_ai/alerts/manager.py`
- [Privacy controls](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/privacy.py): `src/ragnarok_ai/privacy.py`
- [Experiment models](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/ab/models.py): `src/ragnarok_ai/ab/models.py`

### Interfaces and integrations

- [CLI commands](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/cli/main.py): `src/ragnarok_ai/cli/main.py`
- [Result reporters](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/reporters/json.py): `src/ragnarok_ai/reporters/json.py`
- [Console reporter](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/reporters/console.py): `src/ragnarok_ai/reporters/console.py`
- [Framework adapters](https://github.com/2501pr0ject/ragnarok-ai/tree/main/src/ragnarok_ai/adapters/frameworks): `src/ragnarok_ai/adapters/frameworks`
- [LLM adapters](https://github.com/2501pr0ject/ragnarok-ai/tree/main/src/ragnarok_ai/adapters/llm): `src/ragnarok_ai/adapters/llm`
- [Extension protocols](https://github.com/2501pr0ject/ragnarok-ai/blob/main/src/ragnarok_ai/core/protocols.py): `src/ragnarok_ai/core/protocols.py`

### Other components

- **Engineer**
- **RAG pipeline**

## Connections

- Engineer → CLI commands: runs (evidence: `README.md`)
- Engineer → Evaluation runner: invokes (evidence: `README.md`)
- CLI commands → Evaluation runner: dispatches (evidence: `src/ragnarok_ai/cli/main.py`)
- Test set types → Evaluation runner: supplies queries (evidence: `README.md`)
- Evaluation runner → RAG pipeline: queries (evidence: `src/ragnarok_ai/core/evaluate.py`)
- Evaluation runner → Retrieval metrics: computes metrics (evidence: `src/ragnarok_ai/core/evaluate.py`)
- Evaluation runner → Generation evaluator: scores generation (evidence: `README.md`)
- Evaluation runner → Cost tracking: tracks usage (evidence: `src/ragnarok_ai/core/evaluate.py`)
- Evaluation runner → Result reporters: returns results (evidence: `README.md`)
- Result reporters → Console reporter: uses (evidence: `src/ragnarok_ai/reporters/json.py`)
- Generation evaluator → Judge contracts: uses contract (evidence: `src/ragnarok_ai/evaluators/judge.py`)
- HHEM judge → Judge contracts: implements (evidence: `src/ragnarok_ai/judges/hhem.py`)
- HHEM judge → Text segmentation: segments text (evidence: `src/ragnarok_ai/judges/hhem.py`)
- MiniCheck judge → Judge contracts: uses contract (evidence: `src/ragnarok_ai/judges/minicheck.py`)
- Generation evaluator → LLM adapters: uses LLM client (evidence: `src/ragnarok_ai/evaluators/judge.py`)
- Judge calibration → Generation evaluator: evaluates samples (evidence: `src/ragnarok_ai/calibration/calibrator.py`)
- Root-cause diagnosis → Evaluation runner: re-evaluates failures (evidence: `src/ragnarok_ai/diagnosis/diagnostician.py`)
- Test generation → Test set types: produces test sets (evidence: `README.md`)
- Traffic mining → Trace models: mines traces (evidence: `src/ragnarok_ai/mining/miner.py`)
- Dataset I/O → Test set types: loads datasets (evidence: `src/ragnarok_ai/dataset/io.py`)
- Trace collection → Trace models: creates traces (evidence: `src/ragnarok_ai/monitor/client.py`)
- Trace collection → Privacy controls: sanitizes data (evidence: `src/ragnarok_ai/monitor/client.py`)
- Trace store → Trace models: stores records (evidence: `src/ragnarok_ai/monitor/store.py`)
- Drift detection → Trace models: checks baseline (evidence: `src/ragnarok_ai/drift/detector.py`)
- Drift detection → Alert management: raises alerts (evidence: `src/ragnarok_ai/drift/detector.py`)
- A/B analysis → Experiment models: builds reports (evidence: `src/ragnarok_ai/ab/analyzer.py`)
- A/B analysis → Trace models: compares runs (evidence: `src/ragnarok_ai/ab/analyzer.py`)
- RAG pipeline → Framework adapters: integrates through (evidence: `README.md`)
- Evaluation runner → Evaluation provenance: records context
- Evaluation runner → Error types: raises errors (evidence: `src/ragnarok_ai/core/evaluate.py`)
- Framework adapters → Extension protocols: implements interfaces
