fareedkhan-dev/scalable-rag-pipeline

1. Overall shape

This repository is a production-oriented, split-plane RAG platform. The main boundary is between a control plane for request handling and orchestration, and a data plane for heavy AI compute and ingestion. The control plane is CPU-first; the data plane is GPU-heavy and autoscaled.

2. External platform and infrastructure layer

The system is designed for AWS EKS with Terraform provisioning the base cloud stack: VPC, EKS, Aurora Postgres, Redis, S3, IAM/IRSA, and Neo4j-related resources. Karpenter provisions CPU and GPU nodes on demand. Kubernetes manifests and Helm charts under deploy/ and infra/ define how each subsystem is placed and connected.

3. API / control plane

services/api is the main application surface. It is a FastAPI service exposing chat, feedback, upload, and health routes. This is the entry point for user traffic and the place where auth, caching, session/memory state, and agent orchestration are coordinated.

4. Agent orchestration layer

The core chat logic is implemented as a LangGraph-style state machine in services/api/app/agents/. The major nodes are planner, retriever, tool, and responder. The planner decides whether to answer directly, retrieve context, or invoke tools. This is the system’s main reasoning boundary and the point where query rewriting and retrieval strategy selection happen.

5. Retrieval and enhancement path

Retrieval is hybrid: vector search and graph search are both first-class. services/api/app/enhancers/ adds query rewriting and HyDE-style expansion before retrieval. services/api/app/tools/ contains vector search, graph search, web search, calculator, and sandbox tools, which the agent can invoke depending on intent. This creates a layered flow: user query → rewrite/enhance → retrieve → reason → answer.

6. State, cache, and memory

Short-lived and conversational state is split across multiple stores. services/api/app/cache/redis.py handles caching and semantic cache behavior, while services/api/app/memory/postgres.py persists memory/session-related data in Postgres. This separates ephemeral request acceleration from durable user/session state.

7. Model serving boundary

Heavy inference is offloaded to Ray Serve services in deploy/ray/. There are two model services: an embedding service and an LLM service. services/api/app/clients/ray_embed.py and services/api/app/clients/ray_llm.py show that the API does not host models directly; it calls remote Ray endpoints instead. Model configs live in models/, including BGE-M3 embeddings and Llama-based LLMs/rerankers.

8. Knowledge stores

The retrieval layer writes and reads from two complementary databases. Qdrant stores dense embeddings for semantic retrieval, and Neo4j stores entities and relationships for graph retrieval. The ingestion pipeline writes to both, and the chat path queries both through dedicated clients and tool modules.

9. Ingestion pipeline

pipelines/ingestion is the offline data plane. Documents are loaded from PDF/DOCX/HTML, chunked, embedded, and converted into graph triples. The pipeline then upserts vectors into Qdrant and merges nodes/edges into Neo4j. pipelines/jobs/ shows this is intended as a Ray job, with S3 event handling as the trigger path. The primary ingestion flow is S3 → Ray job → parsing/chunking → embedding/graph extraction → persistent indexes.

10. Sandboxed tool execution

services/sandbox is a separate execution boundary for restricted code or computation tasks. It has its own Docker image, limits, and network policy, indicating untrusted or isolated tool execution is intentionally segregated from the main API process.

11. Ingress and edge routing

deploy/ingress/ provides both Kong and NGINX ingress options, showing the API is meant to be exposed behind a Kubernetes ingress layer rather than directly. The gateway component under services/gateway adds rate limiting at the edge, which is a distinct concern from application auth.

12. Security and secrets

Secrets are not hardcoded; the repo includes External Secrets support and environment templates. JWT auth is implemented in services/api/app/auth/, and deployment expects credentials for Postgres, Redis, Neo4j, Qdrant, and Ray endpoints to be injected at runtime. Network policy files in the sandbox reinforce the separation between trusted API code and constrained execution environments.

13. Observability and operational support

libs/observability/ provides metrics and tracing support, while libs/retry/ and libs/utils/ hold shared operational helpers like backoff, IDs, and timing. This indicates the system is designed for distributed tracing and resilient remote calls across Ray, databases, and external services.

14. Validation and evaluation

The eval/ directory contains datasets, judge code, and RAG evaluation scripts, showing the project treats retrieval quality as a measurable subsystem. This is important architecturally because the repo is not only deployment code; it also includes an evaluation loop for retrieval and answer quality.

15. How the main request flow fits together

A typical chat request enters through the ingress/gateway, reaches the FastAPI API, is optionally authenticated and cached, then passes into the agent graph. The agent may call Ray-hosted embedding/LLM services, query Qdrant and Neo4j, consult memory in Postgres, and use Redis for caching before the final response is streamed back to the client.

16. How to draw the diagram

The cleanest diagram is: Edge/Gateway → API/Agent Orchestrator → (Caches + Memory + Retrieval Stores + Tool Sandbox + Ray Model Services) → Response, with a separate Ingestion Plane flowing from S3/Ray job → Parsers/Chunkers → Embeddings/Graph Extraction → Qdrant + Neo4j. Surround both planes with the AWS/EKS/Karpenter/Terraform infrastructure layer.