# devdreamer26/distributed_hybrid_rag_engine architecture

> Architecture diagram of the GitHub repository devdreamer26/distributed_hybrid_rag_engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/devdreamer26/distributed_hybrid_rag_engine
- Repository: https://github.com/devdreamer26/distributed_hybrid_rag_engine

## Overview

This is a distributed document-ingestion and grounded-question-answering platform. Users upload PDFs through the React dashboard; FastAPI records and dispatches ingestion work, Celery extracts and chunks text, embeds it, and indexes chunks for dense and lexical search. Queries pass through hybrid retrieval with rank fusion, cross-encoder reranking, and grounded answer generation. The graph includes the documented Redis queue and Groq integration; source excerpts do not show every infrastructure or frontend/API detail, so edges are limited to visible calls and README-described relationships.

Main entry points: [`backend/app/main.py`](https://github.com/devdreamer26/distributed_hybrid_rag_engine/blob/main/backend/app/main.py) (FastAPI routes), [`backend/app/workers/celery_worker.py`](https://github.com/devdreamer26/distributed_hybrid_rag_engine/blob/main/backend/app/workers/celery_worker.py) (Ingestion worker), [`backend/app/services/search.py`](https://github.com/devdreamer26/distributed_hybrid_rag_engine/blob/main/backend/app/services/search.py) (Hybrid search and RRF), [`frontend/src/App.jsx`](https://github.com/devdreamer26/distributed_hybrid_rag_engine/blob/main/frontend/src/App.jsx) (React dashboard)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_client["User interface"]
  node_dashboard["React dashboard<br/>[App.jsx]"]
end

subgraph group_api["API orchestration"]
  node_api_entry["FastAPI routes<br/>[main.py]"]
end

subgraph group_ingest["Ingestion and indexing"]
  node_parser["PDF parsing and chunking<br/>[parser.py]"]
  node_worker["Ingestion worker<br/>[celery_worker.py]"]
  node_embedding["Embedding service<br/>[embedding.py]"]
end

subgraph group_retrieve["Retrieval and generation"]
  node_hybrid_search["Hybrid search and RRF<br/>[search.py]"]
  node_reranker["Cross-encoder reranker<br/>[reranker.py]"]
  node_generator["Grounded answer generator<br/>[generator.py]"]
end

subgraph group_state["State and messaging"]
  node_schema["Document and chunk models<br/>[schema.py]"]
  node_database_config["Database sessions<br/>[database.py]"]
  node_postgres[("PostgreSQL and pgvector")]
  node_redis["Redis broker"]
end

node_user(("Document user"))
node_groq{{"Groq inference API"}}

node_user -->|"uploads and asks"| node_dashboard
node_dashboard -->|"sends requests"| node_api_entry
node_api_entry -->|"returns status and results"| node_dashboard
node_api_entry -->|"creates and reads records"| node_schema
node_api_entry -->|"dispatches ingestion"| node_worker
node_worker -->|"uses broker"| node_redis
node_worker -->|"extracts and chunks"| node_parser
node_worker -->|"generates batch vectors"| node_embedding
node_worker -->|"creates chunk records"| node_schema
node_worker -->|"writes indexed chunks"| node_postgres
node_api_entry -->|"gets async sessions"| node_database_config
node_database_config -->|"opens sessions"| node_postgres
node_api_entry -->|"requests candidates"| node_hybrid_search
node_hybrid_search -->|"encodes query"| node_embedding
node_hybrid_search -->|"queries vectors and terms"| node_postgres
node_api_entry -->|"reranks candidates"| node_reranker
node_api_entry -->|"synthesizes from ranked chunks"| node_generator
node_generator -->|"requests completion"| node_groq

click node_dashboard "https://github.com/devdreamer26/distributed_hybrid_rag_engine/blob/main/frontend/src/App.jsx"
click node_api_entry "https://github.com/devdreamer26/distributed_hybrid_rag_engine/blob/main/backend/app/main.py"
click node_parser "https://github.com/devdreamer26/distributed_hybrid_rag_engine/blob/main/backend/app/services/parser.py"
click node_worker "https://github.com/devdreamer26/distributed_hybrid_rag_engine/blob/main/backend/app/workers/celery_worker.py"
click node_embedding "https://github.com/devdreamer26/distributed_hybrid_rag_engine/blob/main/backend/app/services/embedding.py"
click node_schema "https://github.com/devdreamer26/distributed_hybrid_rag_engine/blob/main/backend/app/models/schema.py"
click node_database_config "https://github.com/devdreamer26/distributed_hybrid_rag_engine/blob/main/backend/app/core/database.py"
click node_hybrid_search "https://github.com/devdreamer26/distributed_hybrid_rag_engine/blob/main/backend/app/services/search.py"
click node_reranker "https://github.com/devdreamer26/distributed_hybrid_rag_engine/blob/main/backend/app/services/reranker.py"
click node_generator "https://github.com/devdreamer26/distributed_hybrid_rag_engine/blob/main/backend/app/services/generator.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dashboard,node_user toneBlue
class node_api_entry toneAmber
class node_parser,node_worker,node_embedding,node_groq toneMint
class node_hybrid_search,node_reranker,node_generator toneRose
class node_schema,node_database_config,node_postgres,node_redis toneIndigo
```

## Components

### User interface

- [React dashboard](https://github.com/devdreamer26/distributed_hybrid_rag_engine/blob/main/frontend/src/App.jsx): `frontend/src/App.jsx`

### API orchestration

- [FastAPI routes](https://github.com/devdreamer26/distributed_hybrid_rag_engine/blob/main/backend/app/main.py): `backend/app/main.py`

### Ingestion and indexing

- [PDF parsing and chunking](https://github.com/devdreamer26/distributed_hybrid_rag_engine/blob/main/backend/app/services/parser.py): `backend/app/services/parser.py`
- [Ingestion worker](https://github.com/devdreamer26/distributed_hybrid_rag_engine/blob/main/backend/app/workers/celery_worker.py): `backend/app/workers/celery_worker.py`
- [Embedding service](https://github.com/devdreamer26/distributed_hybrid_rag_engine/blob/main/backend/app/services/embedding.py): `backend/app/services/embedding.py`

### Retrieval and generation

- [Hybrid search and RRF](https://github.com/devdreamer26/distributed_hybrid_rag_engine/blob/main/backend/app/services/search.py): `backend/app/services/search.py`
- [Cross-encoder reranker](https://github.com/devdreamer26/distributed_hybrid_rag_engine/blob/main/backend/app/services/reranker.py): `backend/app/services/reranker.py`
- [Grounded answer generator](https://github.com/devdreamer26/distributed_hybrid_rag_engine/blob/main/backend/app/services/generator.py): `backend/app/services/generator.py`

### State and messaging

- [Document and chunk models](https://github.com/devdreamer26/distributed_hybrid_rag_engine/blob/main/backend/app/models/schema.py): `backend/app/models/schema.py`
- [Database sessions](https://github.com/devdreamer26/distributed_hybrid_rag_engine/blob/main/backend/app/core/database.py): `backend/app/core/database.py`
- **PostgreSQL and pgvector**
- **Redis broker**

### Other components

- **Document user**
- **Groq inference API**

## Connections

- Document user → React dashboard: uploads and asks (evidence: `frontend/src/App.jsx`)
- React dashboard → FastAPI routes: sends requests (evidence: `frontend/src/App.jsx`)
- FastAPI routes → React dashboard: returns status and results (evidence: `frontend/src/App.jsx`)
- FastAPI routes → Document and chunk models: creates and reads records (evidence: `backend/app/main.py`)
- FastAPI routes → Ingestion worker: dispatches ingestion (evidence: `backend/app/main.py`)
- Ingestion worker → Redis broker: uses broker (evidence: `backend/app/workers/celery_worker.py`)
- Ingestion worker → PDF parsing and chunking: extracts and chunks (evidence: `backend/app/workers/celery_worker.py`)
- Ingestion worker → Embedding service: generates batch vectors (evidence: `backend/app/workers/celery_worker.py`)
- Ingestion worker → Document and chunk models: creates chunk records (evidence: `backend/app/workers/celery_worker.py`)
- Ingestion worker → PostgreSQL and pgvector: writes indexed chunks (evidence: `backend/app/workers/celery_worker.py`)
- FastAPI routes → Database sessions: gets async sessions (evidence: `backend/app/main.py`)
- Database sessions → PostgreSQL and pgvector: opens sessions (evidence: `backend/app/core/database.py`)
- FastAPI routes → Hybrid search and RRF: requests candidates (evidence: `backend/app/main.py`)
- Hybrid search and RRF → Embedding service: encodes query (evidence: `backend/app/services/search.py`)
- Hybrid search and RRF → PostgreSQL and pgvector: queries vectors and terms (evidence: `backend/app/services/search.py`)
- FastAPI routes → Cross-encoder reranker: reranks candidates (evidence: `backend/app/main.py`)
- FastAPI routes → Grounded answer generator: synthesizes from ranked chunks (evidence: `backend/app/main.py`)
- Grounded answer generator → Groq inference API: requests completion (evidence: `backend/app/services/generator.py`)
