# chetanchandane/rag-data-plane architecture

> Architecture diagram of the GitHub repository chetanchandane/rag-data-plane, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/chetanchandane/rag-data-plane
- Repository: https://github.com/chetanchandane/rag-data-plane

## Overview

This repository implements a document-ingestion data plane: clients register document versions, upload PDFs directly to object storage, and the worker indexes them into tenant-scoped Postgres. The worker validates and claims jobs, verifies and parses PDFs, chunks content, obtains embeddings, and stores/indexes results. The graph includes the API, shared security/data and job-contract layers, worker stages, and the issuer utility. Some capabilities (including reindex/delete job types) are visible in the job contract, but their runtime handling is not established by the excerpts; evidence limits are reflected by omitting those flows.

Main entry points: [`services/worker/handler.py`](https://github.com/chetanchandane/rag-data-plane/blob/main/services/worker/handler.py) (Queue handler), [`services/api/app.py`](https://github.com/chetanchandane/rag-data-plane/blob/main/services/api/app.py) (FastAPI app), [`services/api/documents.py`](https://github.com/chetanchandane/rag-data-plane/blob/main/services/api/documents.py) (Document routes), [`services/worker/pipeline.py`](https://github.com/chetanchandane/rag-data-plane/blob/main/services/worker/pipeline.py) (Ingest pipeline)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Document API"]
  node_api_app["FastAPI app<br/>[app.py]"]
  node_api_auth["Bearer claims<br/>[auth.py]"]
  node_api_config["API settings<br/>[config.py]"]
  node_documents["Document routes<br/>[documents.py]"]
  node_uploads["Upload presigning<br/>[uploads.py]"]
  node_issuer["Token issuer<br/>[issuer.py]"]
end

subgraph group_shared["Shared services"]
  node_tokens["Token utilities<br/>[tokens.py]"]
  node_db_access["DB access and RLS<br/>[db.py]"]
  node_job_contract["Ingest job contract<br/>[jobs.py]"]
  node_errors["Failure classification<br/>[errors.py]"]
  node_telemetry["Tracing<br/>[telemetry.py]"]
end

subgraph group_worker["Ingestion worker"]
  node_handler["Queue handler<br/>[handler.py]"]
  node_pipeline["Ingest pipeline<br/>[pipeline.py]"]
  node_pdf["PDF parsing<br/>[pdf.py]"]
  node_chunking["Chunking<br/>[chunking.py]"]
  node_embedding["Embedding client<br/>[embedding.py]"]
  node_store["Ingest persistence<br/>[store.py]"]
end

subgraph group_external["External systems"]
  node_postgres[("Postgres")]
  node_s3[("S3 object storage")]
  node_queue["SQS and DLQ"]
  node_bedrock["Bedrock embeddings"]
  node_observability["Telemetry backend"]
end

node_client(("API client"))

node_client -->|"calls"| node_api_app
node_api_app -->|"mounts routes"| node_documents
node_api_app -->|"loads settings"| node_api_config
node_api_app -->|"creates pool"| node_db_access
node_api_app -->|"creates S3 client"| node_uploads
node_documents -->|"gets claims"| node_api_auth
node_api_auth -->|"reads settings"| node_api_config
node_documents -->|"queries via RLS"| node_db_access
node_documents -->|"presigns uploads"| node_uploads
node_documents -->|"uses job contract"| node_job_contract
node_client -->|"uploads PDF"| node_s3
node_issuer -->|"uses token utilities"| node_tokens
node_handler -->|"validates messages"| node_job_contract
node_handler -->|"runs ingestion"| node_pipeline
node_handler -->|"creates pool"| node_db_access
node_handler -->|"supplies embedder"| node_embedding
node_handler -->|"classifies failures"| node_errors
node_handler -->|"sets up tracing"| node_telemetry
node_queue -.->|"delivers jobs"| node_handler
node_handler -->|"sends DLQ messages"| node_queue
node_handler -->|"loads source"| node_s3
node_pipeline -->|"verifies and parses"| node_pdf
node_pipeline -->|"chunks document"| node_chunking
node_pipeline -->|"claims and persists"| node_store
node_pipeline -->|"calls embed callback"| node_embedding
node_store -->|"uses RLS transactions"| node_db_access
node_db_access -->|"reads and writes"| node_postgres
node_embedding -->|"invokes model"| node_bedrock
node_telemetry -.->|"exports traces"| node_observability

click node_api_app "https://github.com/chetanchandane/rag-data-plane/blob/main/services/api/app.py"
click node_api_auth "https://github.com/chetanchandane/rag-data-plane/blob/main/services/api/auth.py"
click node_api_config "https://github.com/chetanchandane/rag-data-plane/blob/main/services/api/config.py"
click node_documents "https://github.com/chetanchandane/rag-data-plane/blob/main/services/api/documents.py"
click node_uploads "https://github.com/chetanchandane/rag-data-plane/blob/main/services/api/uploads.py"
click node_issuer "https://github.com/chetanchandane/rag-data-plane/blob/main/scripts/issuer.py"
click node_tokens "https://github.com/chetanchandane/rag-data-plane/blob/main/services/shared/tokens.py"
click node_db_access "https://github.com/chetanchandane/rag-data-plane/blob/main/services/shared/db.py"
click node_job_contract "https://github.com/chetanchandane/rag-data-plane/blob/main/services/shared/jobs.py"
click node_errors "https://github.com/chetanchandane/rag-data-plane/blob/main/services/shared/errors.py"
click node_telemetry "https://github.com/chetanchandane/rag-data-plane/blob/main/services/shared/telemetry.py"
click node_handler "https://github.com/chetanchandane/rag-data-plane/blob/main/services/worker/handler.py"
click node_pipeline "https://github.com/chetanchandane/rag-data-plane/blob/main/services/worker/pipeline.py"
click node_pdf "https://github.com/chetanchandane/rag-data-plane/blob/main/services/worker/pdf.py"
click node_chunking "https://github.com/chetanchandane/rag-data-plane/blob/main/services/worker/chunking.py"
click node_embedding "https://github.com/chetanchandane/rag-data-plane/blob/main/services/worker/embedding.py"
click node_store "https://github.com/chetanchandane/rag-data-plane/blob/main/services/worker/store.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api_app,node_api_auth,node_api_config,node_documents,node_uploads,node_issuer,node_client toneBlue
class node_tokens,node_db_access,node_job_contract,node_errors,node_telemetry toneAmber
class node_handler,node_pipeline,node_pdf,node_chunking,node_embedding,node_store toneMint
class node_postgres,node_s3,node_queue,node_bedrock,node_observability toneRose
```

## Components

### Document API

- [FastAPI app](https://github.com/chetanchandane/rag-data-plane/blob/main/services/api/app.py): `services/api/app.py`
- [Bearer claims](https://github.com/chetanchandane/rag-data-plane/blob/main/services/api/auth.py): `services/api/auth.py`
- [API settings](https://github.com/chetanchandane/rag-data-plane/blob/main/services/api/config.py): `services/api/config.py`
- [Document routes](https://github.com/chetanchandane/rag-data-plane/blob/main/services/api/documents.py): `services/api/documents.py`
- [Upload presigning](https://github.com/chetanchandane/rag-data-plane/blob/main/services/api/uploads.py): `services/api/uploads.py`
- [Token issuer](https://github.com/chetanchandane/rag-data-plane/blob/main/scripts/issuer.py): `scripts/issuer.py`

### Shared services

- [Token utilities](https://github.com/chetanchandane/rag-data-plane/blob/main/services/shared/tokens.py): `services/shared/tokens.py`
- [DB access and RLS](https://github.com/chetanchandane/rag-data-plane/blob/main/services/shared/db.py): `services/shared/db.py`
- [Ingest job contract](https://github.com/chetanchandane/rag-data-plane/blob/main/services/shared/jobs.py): `services/shared/jobs.py`
- [Failure classification](https://github.com/chetanchandane/rag-data-plane/blob/main/services/shared/errors.py): `services/shared/errors.py`
- [Tracing](https://github.com/chetanchandane/rag-data-plane/blob/main/services/shared/telemetry.py): `services/shared/telemetry.py`

### Ingestion worker

- [Queue handler](https://github.com/chetanchandane/rag-data-plane/blob/main/services/worker/handler.py): `services/worker/handler.py`
- [Ingest pipeline](https://github.com/chetanchandane/rag-data-plane/blob/main/services/worker/pipeline.py): `services/worker/pipeline.py`
- [PDF parsing](https://github.com/chetanchandane/rag-data-plane/blob/main/services/worker/pdf.py): `services/worker/pdf.py`
- [Chunking](https://github.com/chetanchandane/rag-data-plane/blob/main/services/worker/chunking.py): `services/worker/chunking.py`
- [Embedding client](https://github.com/chetanchandane/rag-data-plane/blob/main/services/worker/embedding.py): `services/worker/embedding.py`
- [Ingest persistence](https://github.com/chetanchandane/rag-data-plane/blob/main/services/worker/store.py): `services/worker/store.py`

### External systems

- **Postgres**
- **S3 object storage**
- **SQS and DLQ**
- **Bedrock embeddings**
- **Telemetry backend**

### Other components

- **API client**

## Connections

- API client → FastAPI app: calls (evidence: `services/api/app.py`)
- FastAPI app → Document routes: mounts routes (evidence: `services/api/app.py`)
- FastAPI app → API settings: loads settings (evidence: `services/api/app.py`)
- FastAPI app → DB access and RLS: creates pool (evidence: `services/api/app.py`)
- FastAPI app → Upload presigning: creates S3 client (evidence: `services/api/app.py`)
- Document routes → Bearer claims: gets claims (evidence: `services/api/documents.py`)
- Bearer claims → API settings: reads settings (evidence: `services/api/auth.py`)
- Document routes → DB access and RLS: queries via RLS (evidence: `services/api/documents.py`)
- Document routes → Upload presigning: presigns uploads (evidence: `services/api/documents.py`)
- Document routes → Ingest job contract: uses job contract (evidence: `services/api/documents.py`)
- API client → S3 object storage: uploads PDF
- Token issuer → Token utilities: uses token utilities (evidence: `scripts/issuer.py`)
- Queue handler → Ingest job contract: validates messages (evidence: `services/worker/handler.py`)
- Queue handler → Ingest pipeline: runs ingestion (evidence: `services/worker/handler.py`)
- Queue handler → DB access and RLS: creates pool (evidence: `services/worker/handler.py`)
- Queue handler → Embedding client: supplies embedder (evidence: `services/worker/handler.py`)
- Queue handler → Failure classification: classifies failures (evidence: `services/worker/handler.py`)
- Queue handler → Tracing: sets up tracing (evidence: `services/worker/handler.py`)
- SQS and DLQ → Queue handler: delivers jobs
- Queue handler → SQS and DLQ: sends DLQ messages (evidence: `services/worker/handler.py`)
- Queue handler → S3 object storage: loads source (evidence: `services/worker/handler.py`)
- Ingest pipeline → PDF parsing: verifies and parses (evidence: `services/worker/pipeline.py`)
- Ingest pipeline → Chunking: chunks document (evidence: `services/worker/pipeline.py`)
- Ingest pipeline → Ingest persistence: claims and persists (evidence: `services/worker/pipeline.py`)
- Ingest pipeline → Embedding client: calls embed callback (evidence: `services/worker/handler.py`)
- Ingest persistence → DB access and RLS: uses RLS transactions (evidence: `services/worker/store.py`)
- DB access and RLS → Postgres: reads and writes (evidence: `services/shared/db.py`)
- Embedding client → Bedrock embeddings: invokes model (evidence: `services/worker/embedding.py`)
- Tracing → Telemetry backend: exports traces
