How chetanchandane/rag-data-plane works
This repository implements a document-ingestion data plane: clients register document versions, upload PDFs directly to object storage, and the worker indexes them into tenant-scoped Postgres. The worker validates and claims jobs, verifies and parses PDFs, chunks content, obtains embeddings, and stores/indexes results. The graph includes the API, shared security/data and job-contract layers, worker stages, and the issuer utility. Some capabilities (including reindex/delete job types) are visible in the job contract, but their runtime handling is not established by the excerpts; evidence limits are reflected by omitting those flows.
Main entry points: handler.py Queue handler · app.py FastAPI app · documents.py Document routes · pipeline.py Ingest pipeline
Components 23
Document API
- FastAPI app
services/api/app.py - Bearer claims
services/api/auth.py - API settings
services/api/config.py - Document routes
services/api/documents.py - Upload presigning
services/api/uploads.py - Token issuer
scripts/issuer.py
Shared services
- Token utilities
services/shared/tokens.py - DB access and RLS
services/shared/db.py - Ingest job contract
services/shared/jobs.py - Failure classification
services/shared/errors.py - Tracing
services/shared/telemetry.py
Ingestion worker
- Queue handler
services/worker/handler.py - Ingest pipeline
services/worker/pipeline.py - PDF parsing
services/worker/pdf.py - Chunking
services/worker/chunking.py - Embedding client
services/worker/embedding.py - Ingest persistence
services/worker/store.py
External systems
- Postgres
- S3 object storage
- SQS and DLQ
- Bedrock embeddings
- Telemetry backend
Other components
- API client
Connections 29
- API client → FastAPI app: calls
services/api/app.py - FastAPI app → Document routes: mounts routes
services/api/app.py - FastAPI app → API settings: loads settings
services/api/app.py - FastAPI app → DB access and RLS: creates pool
services/api/app.py - FastAPI app → Upload presigning: creates S3 client
services/api/app.py - Document routes → Bearer claims: gets claims
services/api/documents.py - Bearer claims → API settings: reads settings
services/api/auth.py - Document routes → DB access and RLS: queries via RLS
services/api/documents.py - Document routes → Upload presigning: presigns uploads
services/api/documents.py - Document routes → Ingest job contract: uses job contract
services/api/documents.py - API client → S3 object storage: uploads PDF
- Token issuer → Token utilities: uses token utilities
scripts/issuer.py - Queue handler → Ingest job contract: validates messages
services/worker/handler.py - Queue handler → Ingest pipeline: runs ingestion
services/worker/handler.py - Queue handler → DB access and RLS: creates pool
services/worker/handler.py - Queue handler → Embedding client: supplies embedder
services/worker/handler.py - Queue handler → Failure classification: classifies failures
services/worker/handler.py - Queue handler → Tracing: sets up tracing
services/worker/handler.py - SQS and DLQ → Queue handler: delivers jobs
- Queue handler → SQS and DLQ: sends DLQ messages
services/worker/handler.py - Queue handler → S3 object storage: loads source
services/worker/handler.py - Ingest pipeline → PDF parsing: verifies and parses
services/worker/pipeline.py - Ingest pipeline → Chunking: chunks document
services/worker/pipeline.py - Ingest pipeline → Ingest persistence: claims and persists
services/worker/pipeline.py - Ingest pipeline → Embedding client: calls embed callback
services/worker/handler.py - Ingest persistence → DB access and RLS: uses RLS transactions
services/worker/store.py - DB access and RLS → Postgres: reads and writes
services/shared/db.py - Embedding client → Bedrock embeddings: invokes model
services/worker/embedding.py - Tracing → Telemetry backend: exports traces