How devdreamer26/distributed_hybrid_rag_engine works
This is a distributed document-ingestion and grounded-question-answering platform. Users upload PDFs through the React dashboard; FastAPI records and dispatches ingestion work, Celery extracts and chunks text, embeds it, and indexes chunks for dense and lexical search. Queries pass through hybrid retrieval with rank fusion, cross-encoder reranking, and grounded answer generation. The graph includes the documented Redis queue and Groq integration; source excerpts do not show every infrastructure or frontend/API detail, so edges are limited to visible calls and README-described relationships.
Main entry points: main.py FastAPI routes · celery_worker.py Ingestion worker · search.py Hybrid search and RRF · App.jsx React dashboard
Components 14
User interface
- React dashboard
frontend/src/App.jsx
API orchestration
- FastAPI routes
backend/app/main.py
Ingestion and indexing
- PDF parsing and chunking
backend/app/services/parser.py - Ingestion worker
backend/app/workers/celery_worker.py - Embedding service
backend/app/services/embedding.py
Retrieval and generation
- Hybrid search and RRF
backend/app/services/search.py - Cross-encoder reranker
backend/app/services/reranker.py - Grounded answer generator
backend/app/services/generator.py
State and messaging
- Document and chunk models
backend/app/models/schema.py - Database sessions
backend/app/core/database.py - PostgreSQL and pgvector
- Redis broker
Other components
- Document user
- Groq inference API
Connections 18
- Document user → React dashboard: uploads and asks
frontend/src/App.jsx - React dashboard → FastAPI routes: sends requests
frontend/src/App.jsx - FastAPI routes → React dashboard: returns status and results
frontend/src/App.jsx - FastAPI routes → Document and chunk models: creates and reads records
backend/app/main.py - FastAPI routes → Ingestion worker: dispatches ingestion
backend/app/main.py - Ingestion worker → Redis broker: uses broker
backend/app/workers/celery_worker.py - Ingestion worker → PDF parsing and chunking: extracts and chunks
backend/app/workers/celery_worker.py - Ingestion worker → Embedding service: generates batch vectors
backend/app/workers/celery_worker.py - Ingestion worker → Document and chunk models: creates chunk records
backend/app/workers/celery_worker.py - Ingestion worker → PostgreSQL and pgvector: writes indexed chunks
backend/app/workers/celery_worker.py - FastAPI routes → Database sessions: gets async sessions
backend/app/main.py - Database sessions → PostgreSQL and pgvector: opens sessions
backend/app/core/database.py - FastAPI routes → Hybrid search and RRF: requests candidates
backend/app/main.py - Hybrid search and RRF → Embedding service: encodes query
backend/app/services/search.py - Hybrid search and RRF → PostgreSQL and pgvector: queries vectors and terms
backend/app/services/search.py - FastAPI routes → Cross-encoder reranker: reranks candidates
backend/app/main.py - FastAPI routes → Grounded answer generator: synthesizes from ranked chunks
backend/app/main.py - Grounded answer generator → Groq inference API: requests completion
backend/app/services/generator.py