How 2501pr0ject/ragnarok-ai works
RAGnarok-AI is a local-first framework for evaluating RAG systems. The main workflow is to provide a test set and RAG pipeline, run retrieval and generation-quality evaluation (optionally with a local LLM judge), then inspect or export results. Separate capabilities generate or mine test sets, calibrate judges, diagnose failures, and compare or monitor configurations for drift and production health. The graph combines sampled call evidence with README-described capabilities; several feature implementations were not sampled, so their internal connections are omitted.
Main entry points: evaluate.py Evaluation runner · models.py Trace models · judge.py Generation evaluator · types.py Test set types
Components 32
Evaluation workflow
- Test set types
src/ragnarok_ai/core/types.py - Evaluation runner
src/ragnarok_ai/core/evaluate.py - Retrieval metrics
src/ragnarok_ai/evaluators/retrieval.py - Cost tracking
src/ragnarok_ai/cost/tracker.py - Evaluation provenance
src/ragnarok_ai/core/provenance.py - Error types
src/ragnarok_ai/core/exceptions.py
Judging and trust
- Generation evaluator
src/ragnarok_ai/evaluators/judge.py - Judge contracts
src/ragnarok_ai/judges/contract.py - HHEM judge
src/ragnarok_ai/judges/hhem.py - MiniCheck judge
src/ragnarok_ai/judges/minicheck.py - Text segmentation
src/ragnarok_ai/text/segmentation.py - Judge calibration
src/ragnarok_ai/calibration/calibrator.py - Root-cause diagnosis
src/ragnarok_ai/diagnosis/diagnostician.py
Test data
- Test generation
src/ragnarok_ai/generators - Traffic mining
src/ragnarok_ai/mining/miner.py - Dataset I/O
src/ragnarok_ai/dataset/io.py
Production intelligence
- Trace collection
src/ragnarok_ai/monitor/client.py - Trace store
src/ragnarok_ai/monitor/store.py - Trace models
src/ragnarok_ai/monitor/models.py - Drift detection
src/ragnarok_ai/drift/detector.py - A/B analysis
src/ragnarok_ai/ab/analyzer.py - Alert management
src/ragnarok_ai/alerts/manager.py - Privacy controls
src/ragnarok_ai/privacy.py - Experiment models
src/ragnarok_ai/ab/models.py
Interfaces and integrations
- CLI commands
src/ragnarok_ai/cli/main.py - Result reporters
src/ragnarok_ai/reporters/json.py - Console reporter
src/ragnarok_ai/reporters/console.py - Framework adapters
src/ragnarok_ai/adapters/frameworks - LLM adapters
src/ragnarok_ai/adapters/llm - Extension protocols
src/ragnarok_ai/core/protocols.py
Other components
- Engineer
- RAG pipeline
Connections 31
- Engineer → CLI commands: runs
README.md - Engineer → Evaluation runner: invokes
README.md - CLI commands → Evaluation runner: dispatches
src/ragnarok_ai/cli/main.py - Test set types → Evaluation runner: supplies queries
README.md - Evaluation runner → RAG pipeline: queries
src/ragnarok_ai/core/evaluate.py - Evaluation runner → Retrieval metrics: computes metrics
src/ragnarok_ai/core/evaluate.py - Evaluation runner → Generation evaluator: scores generation
README.md - Evaluation runner → Cost tracking: tracks usage
src/ragnarok_ai/core/evaluate.py - Evaluation runner → Result reporters: returns results
README.md - Result reporters → Console reporter: uses
src/ragnarok_ai/reporters/json.py - Generation evaluator → Judge contracts: uses contract
src/ragnarok_ai/evaluators/judge.py - HHEM judge → Judge contracts: implements
src/ragnarok_ai/judges/hhem.py - HHEM judge → Text segmentation: segments text
src/ragnarok_ai/judges/hhem.py - MiniCheck judge → Judge contracts: uses contract
src/ragnarok_ai/judges/minicheck.py - Generation evaluator → LLM adapters: uses LLM client
src/ragnarok_ai/evaluators/judge.py - Judge calibration → Generation evaluator: evaluates samples
src/ragnarok_ai/calibration/calibrator.py - Root-cause diagnosis → Evaluation runner: re-evaluates failures
src/ragnarok_ai/diagnosis/diagnostician.py - Test generation → Test set types: produces test sets
README.md - Traffic mining → Trace models: mines traces
src/ragnarok_ai/mining/miner.py - Dataset I/O → Test set types: loads datasets
src/ragnarok_ai/dataset/io.py - Trace collection → Trace models: creates traces
src/ragnarok_ai/monitor/client.py - Trace collection → Privacy controls: sanitizes data
src/ragnarok_ai/monitor/client.py - Trace store → Trace models: stores records
src/ragnarok_ai/monitor/store.py - Drift detection → Trace models: checks baseline
src/ragnarok_ai/drift/detector.py - Drift detection → Alert management: raises alerts
src/ragnarok_ai/drift/detector.py - A/B analysis → Experiment models: builds reports
src/ragnarok_ai/ab/analyzer.py - A/B analysis → Trace models: compares runs
src/ragnarok_ai/ab/analyzer.py - RAG pipeline → Framework adapters: integrates through
README.md - Evaluation runner → Evaluation provenance: records context
- Evaluation runner → Error types: raises errors
src/ragnarok_ai/core/evaluate.py - Framework adapters → Extension protocols: implements interfaces