How ragpi/ragpi works
Project Type and Purpose
• Ragpi is an API-first backend service (no UI code here) that:
• Ingests documentation, GitHub issues, and READMEs via “connectors.”
• Stores and indexes the ingested content in Redis and/or PostgreSQL.
• Exposes REST endpoints for sources, tasks, healthchecks, and chat.
• Uses Celery for asynchronous sync tasks.
• Routes chat queries through LLM providers (OpenAI, Ollama, Deepseek).
• Offers integration webhooks for Slack, Discord, and a web widget.
File Structure
• src/main.py: entry point (FastAPI app).
• src/chat: chat API router, service layer, prompt definitions.
• src/sources & src/tasks: endpoints and business logic for source creation, sync tasks, and monitoring.
• src/connectors: one directory per connector type (sitemap, GitHub issues, GitHub README), plus base & common helper code.
• src/document_store: pluggable stores (Postgres, Redis) with ranking and schema definitions.
• src/llm_providers: client abstractions, constants, and validators for LLM APIs.
• src/lock: distributed lock service (to prevent concurrent syncs).
• src/celery.py + Dockerfiles + docker-compose.yml: orchestrates async workers, Redis broker, Postgres DB.
• Config & common utilities at top level: logging, OpenTelemetry, exceptions, API key management.
Extract Architectural Insights from README
• API-first design: all functionality exposed via HTTP endpoints.
• Agentic RAG: chat service dynamically retrieves docs at query time.
• Supports multiple providers + integrations.
• Docker + Docker Compose for local/prod deployment.
Main Components
a. API Layer (FastAPI)
• Endpoints: /chat, /sources, /tasks, /healthcheck
• Routers load their services and dependencies
b. Connectors
• Sitemap, GitHub issues, GitHub README
• Fetchers → chunkers → normalized schemas
c. Sync Workers (Celery)
• Broker: Redis
• Worker nodes run tasks from /tasks and sync sources
d. Document Store
• Redis store for fast vector/field lookup
• Postgres store for durable storage and relational data
• Ranking module to score and retrieve docs
e. LLM Providers
• Abstraction layer over OpenAI, Ollama, Deepseek
• Embeddings + completion calls
f. Locking Service
• Ensures single sync per source
g. Monitoring & Observability
• Healthcheck endpoint
• OpenTelemetry instrumentation
h. External Integrations
• Slack, Discord, Web Widget (optional flows)
Relationships & Data Flow
1. Client → FastAPI HTTP → API Layer
2. /sources POST → Connector configuration stored → triggers Celery sync task
3. Celery Worker fetches data via Connector → chunker → writes to Document Store
4. /tasks GET → API Layer reads Celery task status
5. /chat POST → Chat Service:
• Retrieves relevant docs from Document Store
• Builds prompt with user messages + docs
• Calls LLM Provider → returns completion
6. /healthcheck → simple DB/Redis ping
7. Slack/Discord events → forwarded to /chat internally
8. Docker Compose brings up:
• API container
• Celery worker container
• Redis broker & cache
• Postgres database
Architectural Patterns & Principles
• Layered Architecture: routers → services → data stores
• Asynchronous Processing: Celery for long-running sync jobs
• Dependency Injection: FastAPI dependencies for config, DB, Redis
• Pluggable Modules: connectors and document stores follow base interfaces
• API-first: everything exposed as REST endpoints