How airweave-ai/airweave works
Project Type and Purpose
• Airweave is a self-hosted, full-stack context-retrieval platform for AI agents.
• It comprises:
• A React/TypeScript frontend (dashboard/UI)
• A FastAPI Python backend (REST API + business logic)
• Supporting microservices/workers (Temporal workflows, Redis pub/sub)
• Data stores (PostgreSQL for metadata, Qdrant for vector embeddings)
• An optional MCP server (Node.js) and a connector toolkit (Monke)
High-Level Architecture Components
a. Frontend
• React/TypeScript app served via Docker (port 8080)
• Connects to backend REST API for user actions, visualization, and live API docs
b. Backend API
• FastAPI service (port 8001)
• Exposes v1 endpoints: collections, source connections, sync, search, billing, auth
• Implements business logic layers (core/, crud/, schemas/, search/, billing/, analytics/)
c. Worker/Orchestration Layer
• Temporal workers execute long-running sync workflows
• Redis for pub/sub (sync triggers, rate limiting, context caching)
d. Data Stores
• PostgreSQL: metadata tables (collections, users, auth, sync jobs) managed via Alembic
• Qdrant: vector store for semantic embeddings
e. External Integrations
• “platform/sources” connectors for 30+ apps (Slack, GitHub, Google Drive, etc.)
• “integrations” clients (Stripe, Auth0)
• Entity extraction, chunking, embedding using embedders (OpenAI, FastEmbed)
f. SDKs & MCP
• Python/TS SDKs wrap REST calls
• MCP microservice (Node.js) offers an alternative GraphQL-like interface
g. Monke
• Connector-generation CLI/framework (Python + its own frontend/backend)
Interactions and Data Flow
1. User → Frontend (HTTP) → Backend API
2. Backend API → PostgreSQL (CRUD) for metadata
3. Backend API triggers syncs → Redis pub/sub → Temporal workflows
4. Workflows fetch data via connectors → chunk/transform → embed → store in Qdrant
5. User/API queries → Backend search service → Qdrant ● optional reranking → return results
6. SDKs or MCP server also consume the same API or directly interface with the backend
Architectural Patterns & Principles
• Layered architecture: UI / API / Business Logic / Data Access / Storage
• Micro-worker pattern: Temporal + Redis for async/background tasks
• Pluggable connectors: strategy pattern for sources, embedders, destinations
• Multi-tenant via OAuth2, API keys, and per-collection isolation
• Incremental sync via cursors, hashing, rate limiting
Elements
• Clear labels: service names, ports, DB types