# fareedkhan-dev/scalable-rag-pipeline architecture

> Architecture diagram of the GitHub repository fareedkhan-dev/scalable-rag-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fareedkhan-dev/scalable-rag-pipeline
- Repository: https://github.com/fareedkhan-dev/scalable-rag-pipeline

## Overview

1. **Overall shape**
   This repository is a production-oriented, split-plane RAG platform. The main boundary is between a **control plane** for request handling and orchestration, and a **data plane** for heavy AI compute and ingestion. The control plane is CPU-first; the data plane is GPU-heavy and autoscaled.

2. **External platform and infrastructure layer**
   The system is designed for **AWS EKS** with **Terraform** provisioning the base cloud stack: VPC, EKS, Aurora Postgres, Redis, S3, IAM/IRSA, and Neo4j-related resources. **Karpenter** provisions CPU and GPU nodes on demand. Kubernetes manifests and Helm charts under `deploy/` and `infra/` define how each subsystem is placed and connected.

3. **API / control plane**
   `services/api` is the main application surface. It is a **FastAPI** service exposing chat, feedback, upload, and health routes. This is the entry point for user traffic and the place where auth, caching, session/memory state, and agent orchestration are coordinated.

4. **Agent orchestration layer**
   The core chat logic is implemented as a **LangGraph-style state machine** in `services/api/app/agents/`. The major nodes are planner, retriever, tool, and responder. The planner decides whether to answer directly, retrieve context, or invoke tools. This is the system’s main reasoning boundary and the point where query rewriting and retrieval strategy selection happen.

5. **Retrieval and enhancement path**
   Retrieval is hybrid: vector search and graph search are both first-class. `services/api/app/enhancers/` adds query rewriting and HyDE-style expansion before retrieval. `services/api/app/tools/` contains vector search, graph search, web search, calculator, and sandbox tools, which the agent can invoke depending on intent. This creates a layered flow: user query → rewrite/enhance → retrieve → reason → answer.

6. **State, cache, and memory**
   Short-lived and conversational state is split across multiple stores. `services/api/app/cache/redis.py` handles caching and semantic cache behavior, while `services/api/app/memory/postgres.py` persists memory/session-related data in Postgres. This separates ephemeral request acceleration from durable user/session state.

7. **Model serving boundary**
   Heavy inference is offloaded to Ray Serve services in `deploy/ray/`. There are two model services: an **embedding service** and an **LLM service**. `services/api/app/clients/ray_embed.py` and `services/api/app/clients/ray_llm.py` show that the API does not host models directly; it calls remote Ray endpoints instead. Model configs live in `models/`, including BGE-M3 embeddings and Llama-based LLMs/rerankers.

8. **Knowledge stores**
   The retrieval layer writes and reads from two complementary databases. **Qdrant** stores dense embeddings for semantic retrieval, and **Neo4j** stores entities and relationships for graph retrieval. The ingestion pipeline writes to both, and the chat path queries both through dedicated clients and tool modules.

9. **Ingestion pipeline**
   `pipelines/ingestion` is the offline data plane. Documents are loaded from PDF/DOCX/HTML, chunked, embedded, and converted into graph triples. The pipeline then upserts vectors into Qdrant and merges nodes/edges into Neo4j. `pipelines/jobs/` shows this is intended as a Ray job, with S3 event handling as the trigger path. The primary ingestion flow is S3 → Ray job → parsing/chunking → embedding/graph extraction → persistent indexes.

10. **Sandboxed tool execution**
    `services/sandbox` is a separate execution boundary for restricted code or computation tasks. It has its own Docker image, limits, and network policy, indicating untrusted or isolated tool execution is intentionally segregated from the main API process.

11. **Ingress and edge routing**
    `deploy/ingress/` provides both Kong and NGINX ingress options, showing the API is meant to be exposed behind a Kubernetes ingress layer rather than directly. The gateway component under `services/gateway` adds rate limiting at the edge, which is a distinct concern from application auth.

12. **Security and secrets**
    Secrets are not hardcoded; the repo includes **External Secrets** support and environment templates. JWT auth is implemented in `services/api/app/auth/`, and deployment expects credentials for Postgres, Redis, Neo4j, Qdrant, and Ray endpoints to be injected at runtime. Network policy files in the sandbox reinforce the separation between trusted API code and constrained execution environments.

13. **Observability and operational support**
    `libs/observability/` provides metrics and tracing support, while `libs/retry/` and `libs/utils/` hold shared operational helpers like backoff, IDs, and timing. This indicates the system is designed for distributed tracing and resilient remote calls across Ray, databases, and external services.

14. **Validation and evaluation**
    The `eval/` directory contains datasets, judge code, and RAG evaluation scripts, showing the project treats retrieval quality as a measurable subsystem. This is important architecturally because the repo is not only deployment code; it also includes an evaluation loop for retrieval and answer quality.

15. **How the main request flow fits together**
    A typical chat request enters through the ingress/gateway, reaches the FastAPI API, is optionally authenticated and cached, then passes into the agent graph. The agent may call Ray-hosted embedding/LLM services, query Qdrant and Neo4j, consult memory in Postgres, and use Redis for caching before the final response is streamed back to the client.

16. **How to draw the diagram**
    The cleanest diagram is: **Edge/Gateway → API/Agent Orchestrator → (Caches + Memory + Retrieval Stores + Tool Sandbox + Ray Model Services) → Response**, with a separate **Ingestion Plane** flowing from **S3/Ray job → Parsers/Chunkers → Embeddings/Graph Extraction → Qdrant + Neo4j**. Surround both planes with the AWS/EKS/Karpenter/Terraform infrastructure layer.

Main entry points: [`services/api/app/agents`](https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/services/api/app/agents) (Agent Graph), [`services/api`](https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/services/api) (API), [`pipelines/ingestion`](https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/pipelines/ingestion) (Ingestion), [`deploy/ray`](https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/deploy/ray) (Ray Serve)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_edge["Edge"]
  node_nginx_kong["Ingress"]
  node_gateway["Gateway<br/>rate limiting"]
end

subgraph group_control["Control Plane"]
  node_api["API"]
  node_agent_graph{{"Agent Graph<br/>LangGraph orchestration"}}
  node_enhancers["Enhancers<br/>query transform"]
  node_tools["Tools<br/>tooling layer"]
end

subgraph group_data["Data Plane"]
  node_ray_serve["Ray Serve<br/>model serving"]
  node_ingestion["Ingestion<br/>offline pipeline"]
  node_ray_jobs["Ray Jobs<br/>job runtime"]
  node_sandbox["Sandbox<br/>isolated runtime"]
end

subgraph group_storage["Storage"]
  node_cache[("Cache")]
  node_memory[("Memory<br/>Postgres state")]
  node_qdrant[("Qdrant<br/>vector store")]
  node_neo4j[("Neo4j<br/>graph store")]
end

subgraph group_platform["Platform"]
  node_terraform["Terraform<br/>infra provisioning"]
  node_karpenter["Karpenter<br/>node autoscaling"]
end

subgraph group_shared["Shared"]
  node_libs["Shared Libs<br/>common packages"]
  node_models["Model Configs<br/>model specs"]
end

subgraph group_eval["Evaluation"]
  node_evals["Evaluation<br/>RAG evals"]
end

node_nginx_kong -->|"routes through"| node_gateway
node_gateway -->|"forwards"| node_api
node_api -->|"orchestrates"| node_agent_graph
node_agent_graph -->|"rewrites"| node_enhancers
node_agent_graph -->|"invokes"| node_tools
node_agent_graph -->|"checks"| node_cache
node_agent_graph -->|"reads/writes"| node_memory
node_agent_graph -->|"retrieves"| node_qdrant
node_agent_graph -->|"retrieves"| node_neo4j
node_api -->|"calls"| node_ray_serve
node_ingestion -->|"embeds with"| node_ray_serve
node_ingestion -->|"indexes"| node_qdrant
node_ingestion -->|"merges"| node_neo4j
node_ray_jobs -->|"runs"| node_ingestion
node_ray_jobs -.->|"isolates"| node_sandbox
node_sandbox -->|"supports tools"| node_api
node_terraform -.->|"provisions"| node_qdrant
node_terraform -.->|"provisions"| node_neo4j
node_terraform -.->|"provisions"| node_ray_serve
node_karpenter -.->|"autoscale"| node_ray_serve
node_karpenter -.->|"autoscale"| node_ingestion
node_api -->|"uses"| node_libs
node_ingestion -->|"uses"| node_models
node_api -->|"uses"| node_models
node_evals -.->|"measures"| node_qdrant
node_evals -.->|"measures"| node_neo4j

click node_nginx_kong "https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/deploy/ingress"
click node_gateway "https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/services/gateway"
click node_api "https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/services/api"
click node_agent_graph "https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/services/api/app/agents"
click node_enhancers "https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/services/api/app/enhancers"
click node_tools "https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/services/api/app/tools"
click node_cache "https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/services/api/app/cache"
click node_memory "https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/services/api/app/memory"
click node_qdrant "https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/deploy/helm/qdrant"
click node_neo4j "https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/deploy/helm/neo4j"
click node_ray_serve "https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/deploy/ray"
click node_ingestion "https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/pipelines/ingestion"
click node_ray_jobs "https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/pipelines/jobs"
click node_sandbox "https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/services/sandbox"
click node_terraform "https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/infra/terraform"
click node_karpenter "https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/infra/karpenter"
click node_libs "https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/libs"
click node_models "https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/models"
click node_evals "https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/eval"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_nginx_kong,node_gateway,node_evals toneBlue
class node_api,node_agent_graph,node_enhancers,node_tools toneAmber
class node_ray_serve,node_ingestion,node_ray_jobs,node_sandbox toneMint
class node_cache,node_memory,node_qdrant,node_neo4j toneRose
class node_terraform,node_karpenter toneIndigo
class node_libs,node_models toneTeal
```

## Components

### Edge

Ingress and gateway boundary

- [Ingress](https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/deploy/ingress): k8s ingress · Kong and NGINX entry layer · `deploy/ingress`
- [Gateway](https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/services/gateway): rate limiting · Edge rate limiting and request shaping · `services/gateway`

### Control Plane

API and orchestration path

- [API](https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/services/api): FastAPI service · Main chat, upload, feedback, and health surface · `services/api`
- [Agent Graph](https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/services/api/app/agents): LangGraph orchestration · Planner-retriever-tool-responder state machine · `services/api/app/agents`
- [Enhancers](https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/services/api/app/enhancers): query transform · Query rewriting and HyDE expansion before retrieval · `services/api/app/enhancers`
- [Tools](https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/services/api/app/tools): tooling layer · Vector, graph, web, calculator, and sandbox tools · `services/api/app/tools`

### Data Plane

Offline ingestion and model serving

- [Ray Serve](https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/deploy/ray): model serving · Remote embedding and LLM inference endpoints · `deploy/ray`
- [Ingestion](https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/pipelines/ingestion): offline pipeline · Document loading, chunking, embedding, and graph extraction · `pipelines/ingestion`
- [Ray Jobs](https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/pipelines/jobs): job runtime · S3-triggered batch execution path for ingestion · `pipelines/jobs`
- [Sandbox](https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/services/sandbox): isolated runtime · Restricted execution boundary for untrusted tools · `services/sandbox`

### Storage

Persistent retrieval and state stores

- [Cache](https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/services/api/app/cache): Redis cache · Ephemeral cache and semantic cache behavior · `services/api/app/cache`
- [Memory](https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/services/api/app/memory): Postgres state · Durable session and conversational memory · `services/api/app/memory`
- [Qdrant](https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/deploy/helm/qdrant): vector store · Dense embedding index for semantic retrieval · `deploy/helm/qdrant`
- [Neo4j](https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/deploy/helm/neo4j): graph store · Entities and relationships for graph retrieval · `deploy/helm/neo4j`

### Platform

Cloud and Kubernetes foundation

- [Terraform](https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/infra/terraform): infra provisioning · Base AWS stack for VPC, EKS, IAM, RDS, Redis, S3, and Neo4j resources · `infra/terraform`
- [Karpenter](https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/infra/karpenter): node autoscaling · CPU and GPU node provisioning on demand · `infra/karpenter`

### Shared

Reusable libraries and models

- [Shared Libs](https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/libs): common packages · Observability, retry, schemas, utilities, and timing helpers · `libs`
- [Model Configs](https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/models): model specs · Embedding, LLM, and reranker configuration · `models`

### Evaluation

Quality and regression checks

- [Evaluation](https://github.com/fareedkhan-dev/scalable-rag-pipeline/tree/main/eval): RAG evals · Datasets, judges, and regression reporting · `eval`

## Connections

- Ingress → Gateway: routes through · Ingress hands traffic to the edge gateway
- Gateway → API: forwards · Rate-limited user requests reach the API
- API → Agent Graph: orchestrates · Chat requests enter the agent state machine
- Agent Graph → Enhancers: rewrites · Planner may expand or rewrite the query
- Agent Graph → Tools: invokes · Tools are selected by intent and reasoning state
- Agent Graph → Cache: checks · Semantic and response caching accelerate repeat queries
- Agent Graph → Memory: reads/writes · Conversation and session state are persisted
- Agent Graph → Qdrant: retrieves · Vector search supplies semantic context
- Agent Graph → Neo4j: retrieves · Graph search supplies entity and relation context
- API → Ray Serve: calls · API outsources heavy embedding and LLM inference
- Ingestion → Ray Serve: embeds with · Offline pipeline uses the embedding service
- Ingestion → Qdrant: indexes · Chunks are upserted into the vector store
- Ingestion → Neo4j: merges · Extracted triples and entities are written to the graph store
- Ray Jobs → Ingestion: runs · Ray job runtime executes the ingestion pipeline
- Ray Jobs → Sandbox: isolates · Untrusted or bounded tasks can be executed separately
- Sandbox → API: supports tools · Sandboxed execution is exposed as a tool boundary
- Terraform → Qdrant: provisions · Infrastructure defines the vector database stack
- Terraform → Neo4j: provisions · Infrastructure defines graph database resources
- Terraform → Ray Serve: provisions · Cloud stack supports model serving runtime
- Karpenter → Ray Serve: autoscale · GPU nodes expand for model-serving load
- Karpenter → Ingestion: autoscale · CPU nodes expand for batch processing
- API → Shared Libs: uses · Shared observability, retry, schemas, and utilities
- Ingestion → Model Configs: uses · Pipeline relies on embedding and extraction model configs
- API → Model Configs: uses · Runtime model selection is configured separately
- Evaluation → Qdrant: measures · Evaluation assesses retrieval quality against stored indexes
- Evaluation → Neo4j: measures · Evaluation also covers graph-backed retrieval quality