How singh4ashmeet/rag_eval works
RAG_EVAL combines a local RAG pipeline with a hosted evaluation workbench. The core workflow ingests documents into normalized chunks, indexes them for retrieval, and generates grounded answers with citations. Evaluation adds dataset-based scoring, variant comparison, and failure diagnosis; the hosted workbench adds browser access and queued jobs. Edges use the sampled orchestration and documented workflow; hosted internals beyond the excerpts are represented only where the README establishes the capability.
Main entry points: main.py RAG orchestrator · pipeline.py Document ingestion · retrieval Retrieval strategies · evaluate.py Evaluation runner
Components 22
User Interfaces
- Workbench UI
frontend/src/App.tsx - Hosted API
apps/api/main.py - Ingest CLI
apps/cli/ingest.py
RAG Pipeline
- Document ingestion
core/ingest/pipeline.py - Format parsers
core/parsers - Chunking
core/chunking/chunker.py - Normalized documents
core/normalize/schema.py - RAG orchestrator
main.py - Retrieval strategies
core/retrieval - Grounded generation
core/generate/pipeline.py - SQLite vector store
core/storage/sqlite_store.py
Model Services
- Ollama embeddings
core/embed/ollama_embedder.py - LLM providers and routing
core/llm/router.py
Evaluation
- Evaluation datasets
eval/datasets/schema.py - Evaluation runner
eval/runners/evaluate.py - Evaluation metrics
eval/metrics - Variant experiments
eval/runners/experiment.py - Failure diagnostics
eval/diagnostics.py
Hosted Workbench
- Background jobs
apps/worker/tasks.py - Postgres job queue
apps/worker/pg_queue.py
Other components
- User
- Ollama service
Connections 26
- User → Ingest CLI: submits documents
README.md - Ingest CLI → Document ingestion: runs ingestion
apps/cli/ingest.py - User → RAG orchestrator: asks question
README.md - Document ingestion → Format parsers: parses files
core/ingest/pipeline.py - Document ingestion → Chunking: chunks document
core/ingest/pipeline.py - Document ingestion → Normalized documents: produces documents
core/ingest/pipeline.py - RAG orchestrator → Document ingestion: ingests document
main.py - RAG orchestrator → Retrieval strategies: retrieves evidence
main.py - RAG orchestrator → SQLite vector store: indexes chunks
main.py - RAG orchestrator → Grounded generation: generates answer
main.py - Retrieval strategies → Ollama embeddings: embeds queries
core/retrieval/dense.py - Retrieval strategies → SQLite vector store: searches vectors
core/retrieval/dense.py - Grounded generation → Retrieval strategies: retrieves sources
core/generate/pipeline.py - Grounded generation → LLM providers and routing: requests completion
core/generate/pipeline.py - Ollama embeddings → Ollama service: requests embeddings
README.md - LLM providers and routing → Ollama service: requests generation
README.md - User → Workbench UI: uses workbench
README.md - Workbench UI → Hosted API: uses hosted API
- Hosted API → Postgres job queue: queues work
apps/api/main.py - Postgres job queue → Background jobs: dispatches jobs
apps/worker/pg_queue.py - Background jobs → Evaluation runner: runs evaluations
apps/worker/tasks.py - Background jobs → Variant experiments: runs comparisons
apps/worker/tasks.py - Evaluation runner → Evaluation datasets: scores dataset
eval/runners/evaluate.py - Evaluation runner → Evaluation metrics: computes scores
eval/runners/evaluate.py - Variant experiments → Evaluation runner: compares variants
README.md - User → Failure diagnostics: reviews failures
README.md