# skandharaj1307/hybridrag architecture

> Architecture diagram of the GitHub repository skandharaj1307/hybridrag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/skandharaj1307/hybridrag
- Repository: https://github.com/skandharaj1307/hybridrag

## Overview

#### Purpose and Entry Points
This repository implements a retrieval-augmented analysis pipeline over software-quality artifacts: structured metadata plus logs, screenshots/OCR, SRS PDFs, defects, reports, and test cases. Primary operational entry points are `main.py`, `index_qdrant.py`, `query_pipeline.py`, `evaluate_retrieval.py`, and `sync_views.py`.

#### Source Data and Parsing Boundary
Raw corpus material resides under `dataset/structured` and `dataset/unstructured`. Format-specific parsers normalize unstructured inputs before downstream chunking:
- Logs: `parsers/log_parser.py`, `dataset/unstructured/failure_logs`
- SRS PDF: `parsers/pdf_parser.py`, `dataset/unstructured/srs/srs_document.pdf`
- Screenshot/OCR data: `parsers/screenshot_parser.py`, `dataset/unstructured/screenshots`
- Structured records loaded from Supabase: `parsers/supabase_loader.py`

This separates source-specific extraction from retrieval-oriented document construction.

#### Content Chunking and Metadata
Artifact-specific chunkers convert parsed content into retrievable units, preserving the semantic structure appropriate to each source type:
- `chunkers/log_chunker.py`
- `chunkers/srs_chunker.py`
- `chunkers/screenshot_chunker.py`
- `chunkers/defect_chunker.py`
- `chunkers/report_chunker.py`
- `chunkers/testcase_chunker.py`

`metadata/metadata_builder.py` constructs metadata attached to chunks; `utils/id_generator.py` supports stable identifiers. The checked-in `dataset/structured/metadata.json` appears to be a structured metadata artifact.

#### Indexing and Vector Infrastructure
`index_qdrant.py` is the indexing stage that likely embeds chunks and writes them into Qdrant. Vector-store connection management is isolated in `retrieval/qdrant_client_provider.py`; Qdrant querying is implemented in `retrieval/qdrant_retriever.py`.

Embedding is an explicit abstraction in `retrieval/embedder.py`, keeping vector-model selection separate from parsing, chunking, and query orchestration. Qdrant is the material retrieval infrastructure dependency.

#### Hybrid Retrieval and Query Control
Retrieval combines lexical and vector search:
- BM25 path: `retrieval/bm25_retriever.py`
- Dense/Qdrant path: `retrieval/qdrant_retriever.py`
- Combination logic: `retrieval/hybrid_retriever.py`
- Query dispatch/routing: `retrieval/query_router.py`

`query_pipeline.py` is the principal query-time coordinator: it accepts a query, routes or retrieves relevant context, and passes that context to answer-generation or execution stages. `evaluate_retrieval.py` assesses retrieval quality, with question inputs available in `vanilla_rag_questions.txt`.

#### LLM and Structured-Query Boundary
`llm/mistral_service.py` provides the generative-model integration, making Mistral an external inference service/runtime dependency. `llm/sql_executor.py` establishes a distinct path for executing SQL, likely when a routed request requires structured data rather than—or alongside—document retrieval.

This boundary is important: retrieval provides grounded text context, while SQL execution accesses structured information through a controlled executor rather than directly from LLM output.

#### Supabase Data Layer and View Synchronization
Supabase is the structured-data backend. `supabase/migrations/20260801221712_create_context_views.sql` defines database context views, while `sync_views.py` synchronizes or materializes them for pipeline use. `parsers/supabase_loader.py` is the bridge from those database records into the corpus/indexing flow.

Runtime configuration is centralized in `config/config.py`. The project is Python-based, as indicated by `pyproject.toml`, `requirements.txt`, and `.python-version`.

Main entry points: [`retrieval/hybrid_retriever.py`](https://github.com/skandharaj1307/hybridrag/blob/main/retrieval/hybrid_retriever.py) (Hybrid retriever), [`retrieval/query_router.py`](https://github.com/skandharaj1307/hybridrag/blob/main/retrieval/query_router.py) (Query router), [`chunkers/log_chunker.py`](https://github.com/skandharaj1307/hybridrag/blob/main/chunkers/log_chunker.py) (Artifact chunkers), [`metadata/metadata_builder.py`](https://github.com/skandharaj1307/hybridrag/blob/main/metadata/metadata_builder.py) (Chunk metadata builder)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_sources["Corpus & Data"]
  node_raw_corpus["Raw artifact corpus<br/>data sources"]
  node_structured_metadata["Metadata artifact<br/>structured data<br/>[metadata.json]"]
end

subgraph group_ingestion["Ingestion &amp; Indexing"]
  node_source_parsers["Format parsers<br/>parser layer<br/>[log_parser.py]"]
  node_supabase_loader["Supabase loader<br/>database loader<br/>[supabase_loader.py]"]
  node_artifact_chunkers["Artifact chunkers<br/>chunking layer<br/>[log_chunker.py]"]
  node_metadata_builder["Chunk metadata builder<br/>metadata layer"]
  node_indexer{{"Indexing entry point<br/>indexing job<br/>[index_qdrant.py]"}}
end

subgraph group_retrieval["Retrieval &amp; Querying"]
  node_embedder["Embedder<br/>embedding abstraction<br/>[embedder.py]"]
  node_qdrant_client["Qdrant client provider<br/>connection layer"]
  node_qdrant[("Qdrant<br/>vector database")]
  node_bm25["BM25 retriever<br/>lexical retrieval<br/>[bm25_retriever.py]"]
  node_dense_retriever["Qdrant retriever<br/>dense retrieval"]
  node_hybrid_retriever["Hybrid retriever<br/>retrieval fusion"]
  node_query_router["Query router<br/>query dispatch<br/>[query_router.py]"]
  node_query_pipeline{{"Query pipeline<br/>query entry point<br/>[query_pipeline.py]"}}
  node_retrieval_evaluation{{"Retrieval evaluation<br/>evaluation entry point"}}
end

subgraph group_structured["Structured Data &amp; LLM"]
  node_mistral{{"Mistral service<br/>LLM integration<br/>[mistral_service.py]"}}
  node_sql_executor["SQL executor<br/>controlled SQL execution<br/>[sql_executor.py]"]
  node_supabase[("Supabase<br/>structured database")]
  node_context_views["Context views<br/>database views"]
  node_view_sync{{"View synchronization<br/>sync entry point<br/>[sync_views.py]"}}
end

node_raw_corpus -->|"extract"| node_source_parsers
node_structured_metadata -->|"supplies metadata"| node_metadata_builder
node_supabase -->|"loads records"| node_supabase_loader
node_source_parsers -->|"normalized content"| node_artifact_chunkers
node_supabase_loader -->|"structured records"| node_artifact_chunkers
node_artifact_chunkers -->|"chunks"| node_metadata_builder
node_metadata_builder -->|"annotated chunks"| node_indexer
node_indexer -->|"embed chunks"| node_embedder
node_embedder -->|"vectors"| node_qdrant
node_qdrant_client -->|"connects"| node_qdrant
node_qdrant -->|"search index"| node_dense_retriever
node_qdrant_client -->|"provides client"| node_dense_retriever
node_bm25 -->|"lexical results"| node_hybrid_retriever
node_dense_retriever -->|"dense results"| node_hybrid_retriever
node_query_pipeline -->|"dispatch query"| node_query_router
node_query_router -->|"retrieval route"| node_hybrid_retriever
node_hybrid_retriever -->|"grounding context"| node_mistral
node_query_router -->|"structured route"| node_sql_executor
node_sql_executor -->|"controlled queries"| node_supabase
node_context_views -->|"defined in"| node_supabase
node_view_sync -->|"synchronizes"| node_context_views
node_retrieval_evaluation -.->|"evaluates"| node_hybrid_retriever

click node_raw_corpus "https://github.com/skandharaj1307/hybridrag/tree/main/dataset/unstructured"
click node_structured_metadata "https://github.com/skandharaj1307/hybridrag/blob/main/dataset/structured/metadata.json"
click node_source_parsers "https://github.com/skandharaj1307/hybridrag/blob/main/parsers/log_parser.py"
click node_supabase_loader "https://github.com/skandharaj1307/hybridrag/blob/main/parsers/supabase_loader.py"
click node_artifact_chunkers "https://github.com/skandharaj1307/hybridrag/blob/main/chunkers/log_chunker.py"
click node_metadata_builder "https://github.com/skandharaj1307/hybridrag/blob/main/metadata/metadata_builder.py"
click node_indexer "https://github.com/skandharaj1307/hybridrag/blob/main/index_qdrant.py"
click node_embedder "https://github.com/skandharaj1307/hybridrag/blob/main/retrieval/embedder.py"
click node_qdrant_client "https://github.com/skandharaj1307/hybridrag/blob/main/retrieval/qdrant_client_provider.py"
click node_bm25 "https://github.com/skandharaj1307/hybridrag/blob/main/retrieval/bm25_retriever.py"
click node_dense_retriever "https://github.com/skandharaj1307/hybridrag/blob/main/retrieval/qdrant_retriever.py"
click node_hybrid_retriever "https://github.com/skandharaj1307/hybridrag/blob/main/retrieval/hybrid_retriever.py"
click node_query_router "https://github.com/skandharaj1307/hybridrag/blob/main/retrieval/query_router.py"
click node_query_pipeline "https://github.com/skandharaj1307/hybridrag/blob/main/query_pipeline.py"
click node_mistral "https://github.com/skandharaj1307/hybridrag/blob/main/llm/mistral_service.py"
click node_sql_executor "https://github.com/skandharaj1307/hybridrag/blob/main/llm/sql_executor.py"
click node_context_views "https://github.com/skandharaj1307/hybridrag/blob/main/supabase/migrations/20260801221712_create_context_views.sql"
click node_view_sync "https://github.com/skandharaj1307/hybridrag/blob/main/sync_views.py"
click node_retrieval_evaluation "https://github.com/skandharaj1307/hybridrag/blob/main/evaluate_retrieval.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_raw_corpus,node_structured_metadata toneBlue
class node_source_parsers,node_supabase_loader,node_artifact_chunkers,node_metadata_builder,node_indexer toneAmber
class node_embedder,node_qdrant_client,node_qdrant,node_bm25,node_dense_retriever,node_hybrid_retriever,node_query_router,node_query_pipeline,node_retrieval_evaluation toneMint
class node_mistral,node_sql_executor,node_supabase,node_context_views,node_view_sync toneRose
```

## Components

### Corpus & Data

- [Raw artifact corpus](https://github.com/skandharaj1307/hybridrag/tree/main/dataset/unstructured): data sources · Structured and unstructured quality artifacts. · `dataset/unstructured`
- [Metadata artifact](https://github.com/skandharaj1307/hybridrag/blob/main/dataset/structured/metadata.json): structured data · `dataset/structured/metadata.json`

### Ingestion & Indexing

- [Format parsers](https://github.com/skandharaj1307/hybridrag/blob/main/parsers/log_parser.py): parser layer · Normalizes logs, PDFs, screenshots, and database records. · `parsers/log_parser.py`
- [Supabase loader](https://github.com/skandharaj1307/hybridrag/blob/main/parsers/supabase_loader.py): database loader · Bridges structured records into corpus ingestion. · `parsers/supabase_loader.py`
- [Artifact chunkers](https://github.com/skandharaj1307/hybridrag/blob/main/chunkers/log_chunker.py): chunking layer · Creates source-aware retrievable units. · `chunkers/log_chunker.py`
- [Chunk metadata builder](https://github.com/skandharaj1307/hybridrag/blob/main/metadata/metadata_builder.py): metadata layer · `metadata/metadata_builder.py`
- [Indexing entry point](https://github.com/skandharaj1307/hybridrag/blob/main/index_qdrant.py): indexing job · Embeds and writes annotated chunks. · `index_qdrant.py`

### Retrieval & Querying

- [Embedder](https://github.com/skandharaj1307/hybridrag/blob/main/retrieval/embedder.py): embedding abstraction · `retrieval/embedder.py`
- [Qdrant client provider](https://github.com/skandharaj1307/hybridrag/blob/main/retrieval/qdrant_client_provider.py): connection layer · `retrieval/qdrant_client_provider.py`
- **Qdrant**: vector database · Persistent dense-vector retrieval infrastructure.
- [BM25 retriever](https://github.com/skandharaj1307/hybridrag/blob/main/retrieval/bm25_retriever.py): lexical retrieval · `retrieval/bm25_retriever.py`
- [Qdrant retriever](https://github.com/skandharaj1307/hybridrag/blob/main/retrieval/qdrant_retriever.py): dense retrieval · `retrieval/qdrant_retriever.py`
- [Hybrid retriever](https://github.com/skandharaj1307/hybridrag/blob/main/retrieval/hybrid_retriever.py): retrieval fusion · Combines lexical and dense results. · `retrieval/hybrid_retriever.py`
- [Query router](https://github.com/skandharaj1307/hybridrag/blob/main/retrieval/query_router.py): query dispatch · Selects retrieval or structured-query paths. · `retrieval/query_router.py`
- [Query pipeline](https://github.com/skandharaj1307/hybridrag/blob/main/query_pipeline.py): query entry point · Coordinates routing, context retrieval, and response execution. · `query_pipeline.py`
- [Retrieval evaluation](https://github.com/skandharaj1307/hybridrag/blob/main/evaluate_retrieval.py): evaluation entry point · Measures retrieval quality using question inputs. · `evaluate_retrieval.py`

### Structured Data & LLM

- [Mistral service](https://github.com/skandharaj1307/hybridrag/blob/main/llm/mistral_service.py): LLM integration · External generative inference boundary. · `llm/mistral_service.py`
- [SQL executor](https://github.com/skandharaj1307/hybridrag/blob/main/llm/sql_executor.py): controlled SQL execution · Executes structured-data access separately from LLM generation. · `llm/sql_executor.py`
- **Supabase**: structured database · Backend for operational structured records and context views.
- [Context views](https://github.com/skandharaj1307/hybridrag/blob/main/supabase/migrations/20260801221712_create_context_views.sql): database views · `supabase/migrations/20260801221712_create_context_views.sql`
- [View synchronization](https://github.com/skandharaj1307/hybridrag/blob/main/sync_views.py): sync entry point · Synchronizes or materializes context views. · `sync_views.py`

## Connections

- Raw artifact corpus → Format parsers: extract
- Metadata artifact → Chunk metadata builder: supplies metadata
- Supabase → Supabase loader: loads records
- Format parsers → Artifact chunkers: normalized content
- Supabase loader → Artifact chunkers: structured records
- Artifact chunkers → Chunk metadata builder: chunks
- Chunk metadata builder → Indexing entry point: annotated chunks
- Indexing entry point → Embedder: embed chunks
- Embedder → Qdrant: vectors
- Qdrant client provider → Qdrant: connects
- Qdrant → Qdrant retriever: search index
- Qdrant client provider → Qdrant retriever: provides client
- BM25 retriever → Hybrid retriever: lexical results
- Qdrant retriever → Hybrid retriever: dense results
- Query pipeline → Query router: dispatch query
- Query router → Hybrid retriever: retrieval route
- Hybrid retriever → Mistral service: grounding context
- Query router → SQL executor: structured route
- SQL executor → Supabase: controlled queries
- Context views → Supabase: defined in
- View synchronization → Context views: synchronizes
- Retrieval evaluation → Hybrid retriever: evaluates