# ragpi/ragpi architecture

> Architecture diagram of the GitHub repository ragpi/ragpi, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ragpi/ragpi
- Repository: https://github.com/ragpi/ragpi

## Overview

##### Project Type and Purpose
- Ragpi is an API-first backend service (no UI code here) that:
- Ingests documentation, GitHub issues, and READMEs via “connectors.”
- Stores and indexes the ingested content in Redis and/or PostgreSQL.
- Exposes REST endpoints for sources, tasks, healthchecks, and chat.
- Uses Celery for asynchronous sync tasks.
- Routes chat queries through LLM providers (OpenAI, Ollama, Deepseek).
- Offers integration webhooks for Slack, Discord, and a web widget.

##### File Structure
- src/main.py: entry point (FastAPI app).
- src/chat: chat API router, service layer, prompt definitions.
- src/sources & src/tasks: endpoints and business logic for source creation, sync tasks, and monitoring.
- src/connectors: one directory per connector type (sitemap, GitHub issues, GitHub README), plus base & common helper code.
- src/document_store: pluggable stores (Postgres, Redis) with ranking and schema definitions.
- src/llm_providers: client abstractions, constants, and validators for LLM APIs.
- src/lock: distributed lock service (to prevent concurrent syncs).
- src/celery.py + Dockerfiles + docker-compose.yml: orchestrates async workers, Redis broker, Postgres DB.
- Config & common utilities at top level: logging, OpenTelemetry, exceptions, API key management.

##### Extract Architectural Insights from README
- API-first design: all functionality exposed via HTTP endpoints.
- Agentic RAG: chat service dynamically retrieves docs at query time.
- Supports multiple providers + integrations.
- Docker + Docker Compose for local/prod deployment.

##### Main Components
a. API Layer (FastAPI)
- Endpoints: /chat, /sources, /tasks, /healthcheck
- Routers load their services and dependencies
b. Connectors
- Sitemap, GitHub issues, GitHub README
- Fetchers → chunkers → normalized schemas
c. Sync Workers (Celery)
- Broker: Redis
- Worker nodes run tasks from /tasks and sync sources
d. Document Store
- Redis store for fast vector/field lookup
- Postgres store for durable storage and relational data
- Ranking module to score and retrieve docs
e. LLM Providers
- Abstraction layer over OpenAI, Ollama, Deepseek
- Embeddings + completion calls
f. Locking Service
- Ensures single sync per source
g. Monitoring & Observability
- Healthcheck endpoint
- OpenTelemetry instrumentation
h. External Integrations
- Slack, Discord, Web Widget (optional flows)

##### Relationships & Data Flow
1. Client → FastAPI HTTP → API Layer
2. /sources POST → Connector configuration stored → triggers Celery sync task
3. Celery Worker fetches data via Connector → chunker → writes to Document Store
4. /tasks GET → API Layer reads Celery task status
5. /chat POST → Chat Service:
- Retrieves relevant docs from Document Store
- Builds prompt with user messages + docs
- Calls LLM Provider → returns completion
6. /healthcheck → simple DB/Redis ping
7. Slack/Discord events → forwarded to /chat internally
8. Docker Compose brings up:
- API container
- Celery worker container
- Redis broker & cache
- Postgres database

##### Architectural Patterns & Principles
- Layered Architecture: routers → services → data stores
- Asynchronous Processing: Celery for long-running sync jobs
- Dependency Injection: FastAPI dependencies for config, DB, Redis
- Pluggable Modules: connectors and document stores follow base interfaces
- API-first: everything exposed as REST endpoints

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client
    Client("Client"):::external

    %% API Layer
    FastAPIApp("FastAPI App"):::api
    click FastAPIApp "https://github.com/ragpi/ragpi/blob/main/src/main.py"

    subgraph "API Routers" 
        ChatRouter("Chat Router (/chat)"):::api
        click ChatRouter "https://github.com/ragpi/ragpi/blob/main/src/chat/router.py"
        SourcesRouter("Sources Router (/sources)"):::api
        click SourcesRouter "https://github.com/ragpi/ragpi/blob/main/src/sources/router.py"
        TasksRouter("Tasks Router (/tasks)"):::api
        click TasksRouter "https://github.com/ragpi/ragpi/blob/main/src/tasks/router.py"
        HealthRouter("Healthcheck Router (/healthcheck)"):::api
        click HealthRouter "https://github.com/ragpi/ragpi/blob/main/src/healthcheck/router.py"
    end

    %% Services under Routers
    ChatRouter --> ChatService
    ChatService("Chat Service"):::worker
    click ChatService "https://github.com/ragpi/ragpi/blob/main/src/chat/service.py"
    ChatService --> ChatPrompts
    ChatPrompts("Chat Prompts"):::worker
    click ChatPrompts "https://github.com/ragpi/ragpi/blob/main/src/chat/prompts.py"
    ChatService --> ChatSchemas
    ChatSchemas("Chat Schemas"):::worker
    click ChatSchemas "https://github.com/ragpi/ragpi/blob/main/src/chat/schemas.py"

    SourcesRouter --> SourcesService
    SourcesService("Sources Service"):::worker
    click SourcesService "https://github.com/ragpi/ragpi/blob/main/src/sources/service.py"
    SourcesService --> SourcesSchemas
    SourcesSchemas("Sources Schemas"):::worker
    click SourcesSchemas "https://github.com/ragpi/ragpi/blob/main/src/sources/schemas.py"
    SourcesService --> SourcesMetadata
    SourcesMetadata("Sources Metadata Store"):::data
    click SourcesMetadata "https://github.com/ragpi/ragpi/tree/main/src/sources/metadata"

    TasksRouter --> TasksService
    TasksService("Tasks Service"):::worker
    click TasksService "https://github.com/ragpi/ragpi/blob/main/src/tasks/service.py"
    TasksService --> TasksSchemas
    TasksSchemas("Tasks Schemas"):::worker
    click TasksSchemas "https://github.com/ragpi/ragpi/blob/main/src/tasks/schemas.py"
    TasksService --> SyncSource
    SyncSource("Sync Source Task"):::worker
    click SyncSource "https://github.com/ragpi/ragpi/blob/main/src/tasks/sync_source.py"

    %% Celery & Worker
    FastAPIApp -->|enqueues| CeleryApp
    CeleryApp("Celery App"):::worker
    click CeleryApp "https://github.com/ragpi/ragpi/blob/main/src/celery.py"
    CeleryApp --> CeleryWorker
    CeleryWorker("Celery Worker"):::worker

    %% Data Stores
    subgraph "Document Store" 
        RedisStore("Redis Store"):::data
        click RedisStore "https://github.com/ragpi/ragpi/blob/main/src/document_store/redis/store.py"
        PostgresStore("Postgres Store"):::data
        click PostgresStore "https://github.com/ragpi/ragpi/blob/main/src/document_store/postgres/store.py"
        Ranking("Ranking Module"):::data
        click Ranking "https://github.com/ragpi/ragpi/blob/main/src/document_store/ranking.py"
        DocSchemas("Document Schemas"):::data
        click DocSchemas "https://github.com/ragpi/ragpi/blob/main/src/document_store/schemas.py"
    end

    %% Connectors
    subgraph "Connectors (src/connectors)" 
        ConnService("Connector Service"):::worker
        click ConnService "https://github.com/ragpi/ragpi/blob/main/src/connectors/service.py"
        Registry("Connector Registry"):::worker
        click Registry "https://github.com/ragpi/ragpi/blob/main/src/connectors/registry.py"
        BaseConnector("Base Connector"):::worker
        click BaseConnector "https://github.com/ragpi/ragpi/blob/main/src/connectors/base/connector.py"
        Chunker("Common Chunker"):::worker
        click Chunker "https://github.com/ragpi/ragpi/blob/main/src/connectors/common/chunker.py"
        subgraph "GitHub Issues Connector" 
            GHIssues("GH Issues"):::worker
            click GHIssues "https://github.com/ragpi/ragpi/tree/main/src/connectors/github_issues"
        end
        subgraph "GitHub Readme Connector"
            GHReadme("GH Readme"):::worker
            click GHReadme "https://github.com/ragpi/ragpi/tree/main/src/connectors/github_readme"
        end
        subgraph "Sitemap Connector"
            Sitemap("Sitemap"):::worker
            click Sitemap "https://github.com/ragpi/ragpi/tree/main/src/connectors/sitemap"
        end
    end

    %% LLM Providers
    subgraph "LLM Providers (src/llm_providers)"
        LLMClient("LLM Client"):::external
        click LLMClient "https://github.com/ragpi/ragpi/blob/main/src/llm_providers/client.py"
        Constants("Provider Constants"):::external
        click Constants "https://github.com/ragpi/ragpi/blob/main/src/llm_providers/constants.py"
        Validators("Validators"):::external
        click Validators "https://github.com/ragpi/ragpi/blob/main/src/llm_providers/validators.py"
    end

    %% Lock Service
    subgraph "Distributed Lock" 
        LockService("Lock Service"):::worker
        click LockService "https://github.com/ragpi/ragpi/blob/main/src/lock/service.py"
        LockDeps("Lock Dependencies"):::worker
        click LockDeps "https://github.com/ragpi/ragpi/blob/main/src/lock/dependencies.py"
    end

    %% Config & Utilities
    subgraph "Config & Common Utils"
        Config("Config"):::worker
        click Config "https://github.com/ragpi/ragpi/blob/main/src/config.py"
        OTel("OpenTelemetry"):::worker
        click OTel "https://github.com/ragpi/ragpi/blob/main/src/common/opentelemetry.py"
        CommonRedis("Common Redis"):::worker
        click CommonRedis "https://github.com/ragpi/ragpi/blob/main/src/common/redis.py"
        APIKey("API Key Mgmt"):::worker
        click APIKey "https://github.com/ragpi/ragpi/blob/main/src/common/api_key.py"
        Exceptions("Exceptions"):::worker
        click Exceptions "https://github.com/ragpi/ragpi/blob/main/src/common/exceptions.py"
    end

    %% Data Flow
    Client --> FastAPIApp
    FastAPIApp --> ChatRouter
    FastAPIApp --> SourcesRouter
    FastAPIApp --> TasksRouter
    FastAPIApp --> HealthRouter
    ChatService --> RedisStore
    ChatService --> PostgresStore
    ChatService --> Ranking
    ChatService --> LLMClient
    SourcesService --> ConnService
    ConnService --> Registry
    ConnService --> BaseConnector
    ConnService --> Chunker
    BaseConnector --> GHIssues
    BaseConnector --> GHReadme
    BaseConnector --> Sitemap
    CeleryWorker --> ConnService
    CeleryWorker --> LockService
    CeleryWorker --> RedisStore
    CeleryWorker --> PostgresStore
    CeleryWorker --> ConnService
    SrcTasks --> CeleryWorker
    FastAPIApp -->|health ping| RedisStore
    FastAPIApp -->|health ping| PostgresStore

    %% Orchestration
    subgraph "Docker Compose"
        APIContainer("API Container"):::external
        WorkerContainer("Celery Worker Container"):::external
        RedisContainer("Redis Broker & Cache"):::data
        PostgresContainer("Postgres DB"):::data
    end
    APIContainer --> FastAPIApp
    WorkerContainer --> CeleryWorker
    RedisContainer --> RedisBroker
    PostgresContainer --> PostgresDB

    %% Styles
    classDef api fill:#D0E4F5,stroke:#0B70C2,color:#0B70C2;
    classDef worker fill:#DFF5E1,stroke:#4CAF50,color:#4CAF50;
    classDef data fill:#FFE4B3,stroke:#FF9800,color:#BF360C;
    classDef external fill:#E0E0E0,stroke:#9E9E9E,color:#424242;
```
