sameershaikh55/utility-aware-rag-opm

How sameershaikh55/utility-aware-rag-opm works

This repository is an interactive utility-aware RAG evaluator. A user submits a query through the Streamlit dashboard; the OPM engine retrieves corpus evidence, elicits and aggregates grounded model samples, then selects an answer or abstention using expected utility. The dashboard also exposes benchmark telemetry, while the evaluation runner computes calibration and utility metrics. The graph follows the sampled calls and README capabilities; unsampled implementation details are represented only where the README supports them.

Main entry points: opm_engine.py OPM engine · eval.py Benchmark evaluation · app.py Streamlit dashboard · retriever.py Corpus retrieval

Components 15

Query experience

Streamlit dashboard src/app.py

OPM decision pipeline

OPM engine src/opm_engine.py
Corpus retrieval src/retriever.py
Domain corpus data/corpus
Prompt templates src/prompts.py
Versioned prompt prompts/rag_opm/v1.yaml
Groq models

Evaluation and telemetry

Benchmark evaluation src/eval.py
Golden queries data/golden_eval.jsonl
Evaluation reports results

Shared runtime support

Result contracts src/schema.py
Runtime configuration src/config.py
File I/O src/io_utils.py
Structured logging src/logger.py

Other components

User

Connections 21

User → Streamlit dashboard: submits query README.md
Streamlit dashboard → OPM engine: runs query src/app.py
OPM engine → Corpus retrieval: retrieves chunks src/opm_engine.py
Corpus retrieval → Domain corpus: reads corpus README.md
Corpus retrieval → File I/O: reads Markdown src/retriever.py
OPM engine → Prompt templates: loads prompts src/opm_engine.py
Prompt templates → Versioned prompt: loads YAML src/prompts.py
OPM engine → Groq models: requests samples src/opm_engine.py
OPM engine → Result contracts: produces decision src/opm_engine.py
Streamlit dashboard → Result contracts: renders results src/app.py
OPM engine → Runtime configuration: uses settings src/opm_engine.py
Corpus retrieval → Runtime configuration: uses corpus path src/retriever.py
OPM engine → Structured logging: logs events src/opm_engine.py
Prompt templates → Runtime configuration: uses prompt path src/prompts.py
Streamlit dashboard → Benchmark evaluation: runs telemetry README.md
Benchmark evaluation → Golden queries: loads records src/eval.py
Benchmark evaluation → OPM engine: evaluates queries src/eval.py
Benchmark evaluation → Result contracts: validates reports src/eval.py
Benchmark evaluation → File I/O: reads and writes src/eval.py
Benchmark evaluation → Evaluation reports: saves report src/eval.py
Benchmark evaluation → Structured logging: logs report src/eval.py

Drawn by GitDiagram on November 13, 2026 from the repository's file tree, README and source files. Read as Markdown ·