# sairishi-exe/r2r architecture

> Architecture diagram of the GitHub repository sairishi-exe/r2r, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sairishi-exe/r2r
- Repository: https://github.com/sairishi-exe/r2r

## Overview

##### Project Type and Purpose
- This is a full-stack, distributed microservices platform with two published SDKs (Python and JavaScript).
- Core functionality: ingestion, parsing, vector indexing, graph construction, RAG-based retrieval, clustering, and orchestration.
- Provides both a HTTP API (v3) built in Python (FastAPI-style routers), worker services (clustering, unstructured ingestion), and client SDKs to integrate from other applications.

##### Key Architectural Elements from the File Tree
- py/core/main/app.py + routers: the HTTP API gateway exposing REST endpoints.
- py/core/providers & py/core/parsers: adapters for databases, LLMs, file stores, OCR, email, scheduler, ingestion backends.
- py/core/main/orchestration + .providers/orchestration: two orchestration engines (“hatchet” & “simple”) coordinating ingestion/graph workflows.
- services/unstructured & services/clustering: self-contained Docker-packaged microservices for task-specific workloads.
- py/migrations + Alembic: database schema evolution.
- docker/ & deployment/k8s/: Docker Compose and Kubernetes manifests (Postgres + PGVector, RabbitMQ, MinIO/S3, pgAdmin, Nginx).
- js/sdk & py/sdk: JavaScript and Python client libraries that wrap HTTP calls to the main API.
- GitHub Actions workflows for CI/CD, multi-language integration tests, and publications to npm/PyPI.

##### How Components Interact
- Clients (via JS/py SDK or direct HTTP) → API Gateway (FastAPI)
- API Gateway → Services layer (auth, ingestion, graph, retrieval, clustering)
- Services orchestrate tasks by emitting messages to RabbitMQ or scheduling jobs via APScheduler.
- Worker services (clustering, unstructured) subscribe to queues, process tasks, and persist results to PostgreSQL/PGVector.
- Orchestration engine coordinates multistep workflows (ingestion → parsing → embedding → graph building → retrieval).
- External LLM/embedding providers (OpenAI, Anthropic, etc.) are called as needed.
- Object storage (MinIO/S3) holds binary inputs; pgvector stores embeddings.
- SDKs encapsulate authentication, request building, and response parsing.

##### Architectural Patterns & Design Principles
- Hexagonal/Clean architecture: clear separation between domain logic (services, assembly/factory) and infrastructure (providers).
- Event-driven/messaging: RabbitMQ for decoupled worker orchestration.
- Dependency Injection & Factory pattern: builder.py and factory.py assemble service graphs at startup.
- Microservices: clustering and unstructured ingestion run as standalone containers.
- Configuration-as-Code: TOML/YAML for environment and Helm/kustomize overlays.

##### Detail Level
- How the Alembic migration service plugs into the database schema.
- If space allows, annotate CI workflows that build/publish SDKs and run integration tests.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Clients Layer
    subgraph "Clients" 
        direction TB
        JS_SDK["JavaScript SDK"]:::sdk
        Py_SDK["Python SDK"]:::sdk
        HTTP_Client["Direct HTTP"]:::external
    end

    %% API & Orchestration
    subgraph "API & Orchestration Layer"
        direction TB
        APIGateway["API Gateway\nFastAPI (py/core/main/app.py)"]:::api
        subgraph "Routers"
            direction TB
            Routers["v3 Routers (py/core/main/api/v3)"]:::api
        end
        subgraph "Orchestration Engines"
            direction TB
            Hatchet_Ingest["Hatchet Ingestion Workflow"]:::orchestrator
            Hatchet_Graph["Hatchet Graph Workflow"]:::orchestrator
            Simple_Ingest["Simple Ingestion Workflow"]:::orchestrator
            Simple_Graph["Simple Graph Workflow"]:::orchestrator
        end
    end

    %% Service Layer
    subgraph "Service Layer"
        direction TB
        Auth_Service["Auth Service"]:::service
        Ingestion_Service["Ingestion Service"]:::service
        Graph_Service["Graph Service"]:::service
        Retrieval_Service["Retrieval Service"]:::service
        Management_Service["Management Service"]:::service
        Maintenance_Service["Maintenance Service"]:::service
    end

    %% Core Providers & Parsers
    subgraph "Providers & Parsers"
        direction TB
        subgraph "Provider Adapters"
            direction TB
            DB_Providers["Database Providers"]:::infra
            LLM_Providers["LLM Providers"]:::infra
            Embeddings_Providers["Embeddings Providers"]:::infra
            File_Providers["File Store Providers"]:::infra
            OCR_Providers["OCR Providers"]:::infra
            Email_Providers["Email Providers"]:::infra
            Auth_Providers["Auth Providers"]:::infra
            Scheduler_Providers["Scheduler (APScheduler)"]:::infra
        end
        subgraph "Parsers"
            direction TB
            Media_Parsers["Media Parsers"]:::infra
            Structured_Parsers["Structured Parsers"]:::infra
            Text_Parsers["Text Parsers"]:::infra
        end
    end

    %% Messaging & Workers
    subgraph "Messaging & Workers"
        direction TB
        Broker["RabbitMQ"]:::broker
        subgraph "Worker Services"
            direction TB
            Clustering_Worker["Clustering Service"]:::worker
            Unstructured_Worker["Unstructured Ingestion Service"]:::worker
        end
    end

    %% Data Stores
    subgraph "Data Stores"
        direction TB
        Postgres["PostgreSQL + pgvector"]:::database
        Object_Store["MinIO / S3"]:::database
    end

    %% External Services
    subgraph "External Services"
        direction TB
        External_LLM["External LLM / Embedding Providers"]:::external
    end

    %% Deployment & CI/CD
    subgraph "Deployment & CI/CD"
        direction TB
        Docker_Compose["Docker Compose"]:::deploy
        K8s_Manifests["Kubernetes Manifests"]:::deploy
        CI_CD["GitHub Actions"]:::cicd
    end

    %% Connections
    JS_SDK -->|"HTTP (REST)"| APIGateway
    Py_SDK -->|"HTTP (REST)"| APIGateway
    HTTP_Client -->|"HTTP (REST)"| APIGateway

    APIGateway -->|"/api/v3 calls"| Routers
    Routers -->|calls| Auth_Service
    Routers -->|calls| Ingestion_Service
    Routers -->|calls| Graph_Service
    Routers -->|calls| Retrieval_Service
    Routers -->|calls| Management_Service
    Routers -->|calls| Maintenance_Service

    Ingestion_Service -->|schedule via APScheduler| Scheduler_Providers
    Ingestion_Service -->|emit AMQP| Broker
    Graph_Service -->|emit AMQP| Broker
    Retrieval_Service -->|SQL queries| Postgres

    Broker -->|AMQP| Clustering_Worker
    Broker -->|AMQP| Unstructured_Worker

    Clustering_Worker -->|SQL writes| Postgres
    Unstructured_Worker -->|binary upload| Object_Store
    Unstructured_Worker -->|SQL writes| Postgres

    Ingestion_Service -->|use| Parser -->|process| Media_Parsers
    Ingestion_Service -->|use| Parser -->|process| Structured_Parsers
    Ingestion_Service -->|use| Parser -->|process| Text_Parsers

    ServiceLayer_Providers["Service Layer"] -.-> DB_Providers
    ServiceLayer_Providers -.-> LLM_Providers
    ServiceLayer_Providers -.-> Embeddings_Providers
    ServiceLayer_Providers -.-> File_Providers
    ServiceLayer_Providers -.-> OCR_Providers
    ServiceLayer_Providers -.-> Email_Providers
    ServiceLayer_Providers -.-> Auth_Providers

    LLM_Providers -->|API calls| External_LLM
    Embeddings_Providers -->|API calls| External_LLM
    File_Providers -->|S3 API| Object_Store

    %% Orchestration flows
    Ingestion_Service -->|orchestrate| Hatchet_Ingest
    Graph_Service -->|orchestrate| Hatchet_Graph
    Ingestion_Service -->|orchestrate| Simple_Ingest
    Graph_Service -->|orchestrate| Simple_Graph

    %% Deployment & CI/CD connections
    Docker_Compose -->|runs| APIGateway
    Docker_Compose -->|runs| Clustering_Worker
    Docker_Compose -->|runs| Unstructured_Worker
    Docker_Compose -->|runs| RabbitMQ
    Docker_Compose -->|runs| Postgres
    Docker_Compose -->|runs| MinIO

    K8s_Manifests -->|deploys| APIGateway
    K8s_Manifests -->|deploys| Clustering_Worker
    K8s_Manifests -->|deploys| Unstructured_Worker
    K8s_Manifests -->|deploys| RabbitMQ
    K8s_Manifests -->|deploys| Postgres
    K8s_Manifests -->|deploys| MinIO

    CI_CD -->|build & test| JS_SDK
    CI_CD -->|build & test| Py_SDK
    CI_CD -->|publish| js/sdk
    CI_CD -->|publish| py/sdk
    CI_CD -->|build| Clustering_Worker
    CI_CD -->|build| Unstructured_Worker

    %% Click Events
    click APIGateway "https://github.com/sairishi-exe/r2r/blob/main/py/core/main/app.py"
    click Routers "https://github.com/sairishi-exe/r2r/tree/main/py/core/main/api/v3/"
    click Hatchet_Ingest "https://github.com/sairishi-exe/r2r/blob/main/py/core/main/orchestration/hatchet/ingestion_workflow.py"
    click Hatchet_Graph "https://github.com/sairishi-exe/r2r/blob/main/py/core/main/orchestration/hatchet/graph_workflow.py"
    click Simple_Ingest "https://github.com/sairishi-exe/r2r/blob/main/py/core/main/orchestration/simple/ingestion_workflow.py"
    click Simple_Graph "https://github.com/sairishi-exe/r2r/blob/main/py/core/main/orchestration/simple/graph_workflow.py"
    click DB_Providers "https://github.com/sairishi-exe/r2r/tree/main/py/core/providers/database/"
    click LLM_Providers "https://github.com/sairishi-exe/r2r/tree/main/py/core/providers/llm/"
    click Embeddings_Providers "https://github.com/sairishi-exe/r2r/tree/main/py/core/providers/embeddings/"
    click File_Providers "https://github.com/sairishi-exe/r2r/tree/main/py/core/providers/file/"
    click OCR_Providers "https://github.com/sairishi-exe/r2r/tree/main/py/core/providers/ocr/"
    click Email_Providers "https://github.com/sairishi-exe/r2r/tree/main/py/core/providers/email/"
    click Auth_Providers "https://github.com/sairishi-exe/r2r/tree/main/py/core/providers/auth/"
    click Scheduler_Providers "https://github.com/sairishi-exe/r2r/blob/main/py/core/providers/scheduler/apscheduler.py"
    click Media_Parsers "https://github.com/sairishi-exe/r2r/tree/main/py/core/parsers/media/"
    click Structured_Parsers "https://github.com/sairishi-exe/r2r/tree/main/py/core/parsers/structured/"
    click Text_Parsers "https://github.com/sairishi-exe/r2r/tree/main/py/core/parsers/text/"
    click Clustering_Worker "https://github.com/sairishi-exe/r2r/blob/main/services/clustering/main.py"
    click Unstructured_Worker "https://github.com/sairishi-exe/r2r/blob/main/services/unstructured/main.py"
    click Postgres "https://github.com/sairishi-exe/r2r/tree/main/py/migrations/"
    click Docker_Compose "https://github.com/sairishi-exe/r2r/blob/main/docker/compose.yaml"
    click K8s_Manifests "https://github.com/sairishi-exe/r2r/tree/main/deployment/k8s/manifests/"
    click JS_SDK "https://github.com/sairishi-exe/r2r/tree/main/js/sdk/"
    click Py_SDK "https://github.com/sairishi-exe/r2r/tree/main/py/sdk/"

    %% Styles
    classDef api fill:#FFA500,stroke:#333,stroke-width:1px
    classDef orchestrator fill:#FFB347,stroke:#333,stroke-width:1px
    classDef service fill:#FFCC80,stroke:#333,stroke-width:1px
    classDef infra fill:#E0E0E0,stroke:#333,stroke-width:1px
    classDef worker fill:#87CEFA,stroke:#333,stroke-width:1px
    classDef database fill:#90EE90,stroke:#333,stroke-width:1px
    classDef broker fill:#FFD700,stroke:#333,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef sdk fill:#D8BFD8,stroke:#333,stroke-width:1px
    classDef deploy fill:#F0F0F0,stroke:#333,stroke-width:1px
    classDef cicd fill:#DEB887,stroke:#333,stroke-width:1px
```
