How sairishi-exe/r2r works
Project Type and Purpose
• This is a full-stack, distributed microservices platform with two published SDKs (Python and JavaScript).
• Core functionality: ingestion, parsing, vector indexing, graph construction, RAG-based retrieval, clustering, and orchestration.
• Provides both a HTTP API (v3) built in Python (FastAPI-style routers), worker services (clustering, unstructured ingestion), and client SDKs to integrate from other applications.
Key Architectural Elements from the File Tree
• py/core/main/app.py + routers: the HTTP API gateway exposing REST endpoints.
• py/core/providers & py/core/parsers: adapters for databases, LLMs, file stores, OCR, email, scheduler, ingestion backends.
• py/core/main/orchestration + .providers/orchestration: two orchestration engines (“hatchet” & “simple”) coordinating ingestion/graph workflows.
• services/unstructured & services/clustering: self-contained Docker-packaged microservices for task-specific workloads.
• py/migrations + Alembic: database schema evolution.
• docker/ & deployment/k8s/: Docker Compose and Kubernetes manifests (Postgres + PGVector, RabbitMQ, MinIO/S3, pgAdmin, Nginx).
• js/sdk & py/sdk: JavaScript and Python client libraries that wrap HTTP calls to the main API.
• GitHub Actions workflows for CI/CD, multi-language integration tests, and publications to npm/PyPI.
How Components Interact
• Clients (via JS/py SDK or direct HTTP) → API Gateway (FastAPI)
• API Gateway → Services layer (auth, ingestion, graph, retrieval, clustering)
• Services orchestrate tasks by emitting messages to RabbitMQ or scheduling jobs via APScheduler.
• Worker services (clustering, unstructured) subscribe to queues, process tasks, and persist results to PostgreSQL/PGVector.
• Orchestration engine coordinates multistep workflows (ingestion → parsing → embedding → graph building → retrieval).
• External LLM/embedding providers (OpenAI, Anthropic, etc.) are called as needed.
• Object storage (MinIO/S3) holds binary inputs; pgvector stores embeddings.
• SDKs encapsulate authentication, request building, and response parsing.
Architectural Patterns & Design Principles
• Hexagonal/Clean architecture: clear separation between domain logic (services, assembly/factory) and infrastructure (providers).
• Event-driven/messaging: RabbitMQ for decoupled worker orchestration.
• Dependency Injection & Factory pattern: builder.py and factory.py assemble service graphs at startup.
• Microservices: clustering and unstructured ingestion run as standalone containers.
• Configuration-as-Code: TOML/YAML for environment and Helm/kustomize overlays.
Detail Level
• How the Alembic migration service plugs into the database schema.
• If space allows, annotate CI workflows that build/publish SDKs and run integration tests.