# singh4ashmeet/rag_eval architecture

> Architecture diagram of the GitHub repository singh4ashmeet/rag_eval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/singh4ashmeet/rag_eval
- Repository: https://github.com/singh4ashmeet/rag_eval

## Overview

RAG_EVAL combines a local RAG pipeline with a hosted evaluation workbench. The core workflow ingests documents into normalized chunks, indexes them for retrieval, and generates grounded answers with citations. Evaluation adds dataset-based scoring, variant comparison, and failure diagnosis; the hosted workbench adds browser access and queued jobs. Edges use the sampled orchestration and documented workflow; hosted internals beyond the excerpts are represented only where the README establishes the capability.

Main entry points: [`main.py`](https://github.com/singh4ashmeet/rag_eval/blob/main/main.py) (RAG orchestrator), [`core/ingest/pipeline.py`](https://github.com/singh4ashmeet/rag_eval/blob/main/core/ingest/pipeline.py) (Document ingestion), [`core/retrieval`](https://github.com/singh4ashmeet/rag_eval/tree/main/core/retrieval) (Retrieval strategies), [`eval/runners/evaluate.py`](https://github.com/singh4ashmeet/rag_eval/blob/main/eval/runners/evaluate.py) (Evaluation runner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["User Interfaces"]
  node_frontend["Workbench UI<br/>[App.tsx]"]
  node_api["Hosted API<br/>[main.py]"]
  node_cli["Ingest CLI<br/>[ingest.py]"]
end

subgraph group_rag["RAG Pipeline"]
  node_ingest["Document ingestion<br/>[pipeline.py]"]
  node_parsers["Format parsers"]
  node_chunking["Chunking<br/>[chunker.py]"]
  node_normalized["Normalized documents<br/>[schema.py]"]
  node_orchestrator["RAG orchestrator<br/>[main.py]"]
  node_retrieval["Retrieval strategies"]
  node_generation["Grounded generation<br/>[pipeline.py]"]
  node_storage[("SQLite vector store<br/>[sqlite_store.py]")]
end

subgraph group_models["Model Services"]
  node_embedder["Ollama embeddings<br/>[ollama_embedder.py]"]
  node_llm["LLM providers and routing<br/>[router.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_dataset["Evaluation datasets<br/>[schema.py]"]
  node_runner["Evaluation runner<br/>[evaluate.py]"]
  node_metrics["Evaluation metrics"]
  node_experiments["Variant experiments<br/>[experiment.py]"]
  node_diagnostics["Failure diagnostics<br/>[diagnostics.py]"]
end

subgraph group_hosted["Hosted Workbench"]
  node_worker["Background jobs<br/>[tasks.py]"]
  node_queue["Postgres job queue<br/>[pg_queue.py]"]
end

node_user(("User"))
node_ollama{{"Ollama service"}}

node_user -->|"submits documents"| node_cli
node_cli -->|"runs ingestion"| node_ingest
node_user -->|"asks question"| node_orchestrator
node_ingest -->|"parses files"| node_parsers
node_ingest -->|"chunks document"| node_chunking
node_ingest -->|"produces documents"| node_normalized
node_orchestrator -->|"ingests document"| node_ingest
node_orchestrator -->|"retrieves evidence"| node_retrieval
node_orchestrator -->|"indexes chunks"| node_storage
node_orchestrator -->|"generates answer"| node_generation
node_retrieval -->|"embeds queries"| node_embedder
node_retrieval -->|"searches vectors"| node_storage
node_generation -->|"retrieves sources"| node_retrieval
node_generation -->|"requests completion"| node_llm
node_embedder -->|"requests embeddings"| node_ollama
node_llm -->|"requests generation"| node_ollama
node_user -->|"uses workbench"| node_frontend
node_frontend -.->|"uses hosted API"| node_api
node_api -->|"queues work"| node_queue
node_queue -->|"dispatches jobs"| node_worker
node_worker -->|"runs evaluations"| node_runner
node_worker -->|"runs comparisons"| node_experiments
node_runner -->|"scores dataset"| node_dataset
node_runner -->|"computes scores"| node_metrics
node_experiments -->|"compares variants"| node_runner
node_user -->|"reviews failures"| node_diagnostics

click node_frontend "https://github.com/singh4ashmeet/rag_eval/blob/main/frontend/src/App.tsx"
click node_api "https://github.com/singh4ashmeet/rag_eval/blob/main/apps/api/main.py"
click node_cli "https://github.com/singh4ashmeet/rag_eval/blob/main/apps/cli/ingest.py"
click node_ingest "https://github.com/singh4ashmeet/rag_eval/blob/main/core/ingest/pipeline.py"
click node_parsers "https://github.com/singh4ashmeet/rag_eval/tree/main/core/parsers"
click node_chunking "https://github.com/singh4ashmeet/rag_eval/blob/main/core/chunking/chunker.py"
click node_normalized "https://github.com/singh4ashmeet/rag_eval/blob/main/core/normalize/schema.py"
click node_orchestrator "https://github.com/singh4ashmeet/rag_eval/blob/main/main.py"
click node_retrieval "https://github.com/singh4ashmeet/rag_eval/tree/main/core/retrieval"
click node_generation "https://github.com/singh4ashmeet/rag_eval/blob/main/core/generate/pipeline.py"
click node_storage "https://github.com/singh4ashmeet/rag_eval/blob/main/core/storage/sqlite_store.py"
click node_embedder "https://github.com/singh4ashmeet/rag_eval/blob/main/core/embed/ollama_embedder.py"
click node_llm "https://github.com/singh4ashmeet/rag_eval/blob/main/core/llm/router.py"
click node_dataset "https://github.com/singh4ashmeet/rag_eval/blob/main/eval/datasets/schema.py"
click node_runner "https://github.com/singh4ashmeet/rag_eval/blob/main/eval/runners/evaluate.py"
click node_metrics "https://github.com/singh4ashmeet/rag_eval/tree/main/eval/metrics"
click node_experiments "https://github.com/singh4ashmeet/rag_eval/blob/main/eval/runners/experiment.py"
click node_diagnostics "https://github.com/singh4ashmeet/rag_eval/blob/main/eval/diagnostics.py"
click node_worker "https://github.com/singh4ashmeet/rag_eval/blob/main/apps/worker/tasks.py"
click node_queue "https://github.com/singh4ashmeet/rag_eval/blob/main/apps/worker/pg_queue.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_frontend,node_api,node_cli,node_user toneBlue
class node_ingest,node_parsers,node_chunking,node_normalized,node_orchestrator,node_retrieval,node_generation,node_storage toneAmber
class node_embedder,node_llm toneMint
class node_dataset,node_runner,node_metrics,node_experiments,node_diagnostics toneRose
class node_worker,node_queue,node_ollama toneIndigo
```

## Components

### User Interfaces

- [Workbench UI](https://github.com/singh4ashmeet/rag_eval/blob/main/frontend/src/App.tsx): `frontend/src/App.tsx`
- [Hosted API](https://github.com/singh4ashmeet/rag_eval/blob/main/apps/api/main.py): `apps/api/main.py`
- [Ingest CLI](https://github.com/singh4ashmeet/rag_eval/blob/main/apps/cli/ingest.py): `apps/cli/ingest.py`

### RAG Pipeline

- [Document ingestion](https://github.com/singh4ashmeet/rag_eval/blob/main/core/ingest/pipeline.py): `core/ingest/pipeline.py`
- [Format parsers](https://github.com/singh4ashmeet/rag_eval/tree/main/core/parsers): `core/parsers`
- [Chunking](https://github.com/singh4ashmeet/rag_eval/blob/main/core/chunking/chunker.py): `core/chunking/chunker.py`
- [Normalized documents](https://github.com/singh4ashmeet/rag_eval/blob/main/core/normalize/schema.py): `core/normalize/schema.py`
- [RAG orchestrator](https://github.com/singh4ashmeet/rag_eval/blob/main/main.py): `main.py`
- [Retrieval strategies](https://github.com/singh4ashmeet/rag_eval/tree/main/core/retrieval): `core/retrieval`
- [Grounded generation](https://github.com/singh4ashmeet/rag_eval/blob/main/core/generate/pipeline.py): `core/generate/pipeline.py`
- [SQLite vector store](https://github.com/singh4ashmeet/rag_eval/blob/main/core/storage/sqlite_store.py): `core/storage/sqlite_store.py`

### Model Services

- [Ollama embeddings](https://github.com/singh4ashmeet/rag_eval/blob/main/core/embed/ollama_embedder.py): `core/embed/ollama_embedder.py`
- [LLM providers and routing](https://github.com/singh4ashmeet/rag_eval/blob/main/core/llm/router.py): `core/llm/router.py`

### Evaluation

- [Evaluation datasets](https://github.com/singh4ashmeet/rag_eval/blob/main/eval/datasets/schema.py): `eval/datasets/schema.py`
- [Evaluation runner](https://github.com/singh4ashmeet/rag_eval/blob/main/eval/runners/evaluate.py): `eval/runners/evaluate.py`
- [Evaluation metrics](https://github.com/singh4ashmeet/rag_eval/tree/main/eval/metrics): `eval/metrics`
- [Variant experiments](https://github.com/singh4ashmeet/rag_eval/blob/main/eval/runners/experiment.py): `eval/runners/experiment.py`
- [Failure diagnostics](https://github.com/singh4ashmeet/rag_eval/blob/main/eval/diagnostics.py): `eval/diagnostics.py`

### Hosted Workbench

- [Background jobs](https://github.com/singh4ashmeet/rag_eval/blob/main/apps/worker/tasks.py): `apps/worker/tasks.py`
- [Postgres job queue](https://github.com/singh4ashmeet/rag_eval/blob/main/apps/worker/pg_queue.py): `apps/worker/pg_queue.py`

### Other components

- **User**
- **Ollama service**

## Connections

- User → Ingest CLI: submits documents (evidence: `README.md`)
- Ingest CLI → Document ingestion: runs ingestion (evidence: `apps/cli/ingest.py`)
- User → RAG orchestrator: asks question (evidence: `README.md`)
- Document ingestion → Format parsers: parses files (evidence: `core/ingest/pipeline.py`)
- Document ingestion → Chunking: chunks document (evidence: `core/ingest/pipeline.py`)
- Document ingestion → Normalized documents: produces documents (evidence: `core/ingest/pipeline.py`)
- RAG orchestrator → Document ingestion: ingests document (evidence: `main.py`)
- RAG orchestrator → Retrieval strategies: retrieves evidence (evidence: `main.py`)
- RAG orchestrator → SQLite vector store: indexes chunks (evidence: `main.py`)
- RAG orchestrator → Grounded generation: generates answer (evidence: `main.py`)
- Retrieval strategies → Ollama embeddings: embeds queries (evidence: `core/retrieval/dense.py`)
- Retrieval strategies → SQLite vector store: searches vectors (evidence: `core/retrieval/dense.py`)
- Grounded generation → Retrieval strategies: retrieves sources (evidence: `core/generate/pipeline.py`)
- Grounded generation → LLM providers and routing: requests completion (evidence: `core/generate/pipeline.py`)
- Ollama embeddings → Ollama service: requests embeddings (evidence: `README.md`)
- LLM providers and routing → Ollama service: requests generation (evidence: `README.md`)
- User → Workbench UI: uses workbench (evidence: `README.md`)
- Workbench UI → Hosted API: uses hosted API
- Hosted API → Postgres job queue: queues work (evidence: `apps/api/main.py`)
- Postgres job queue → Background jobs: dispatches jobs (evidence: `apps/worker/pg_queue.py`)
- Background jobs → Evaluation runner: runs evaluations (evidence: `apps/worker/tasks.py`)
- Background jobs → Variant experiments: runs comparisons (evidence: `apps/worker/tasks.py`)
- Evaluation runner → Evaluation datasets: scores dataset (evidence: `eval/runners/evaluate.py`)
- Evaluation runner → Evaluation metrics: computes scores (evidence: `eval/runners/evaluate.py`)
- Variant experiments → Evaluation runner: compares variants (evidence: `README.md`)
- User → Failure diagnostics: reviews failures (evidence: `README.md`)
