How ki-quadrat/knowledgebase-backend works
1. Overall shape
- This is a Python 3.13 FastAPI backend called the “data plane.”
- It sits behind /api/v1 and exposes two broad surfaces: shared APIs for health/search/classification/metrics and online APIs for scraping, parsing, ingesting, and vector management.
2. Request entrypoint and app wiring
- app/main.py is the composition root: it creates the FastAPI app, wires lifespan/startup behavior, registers middleware, and mounts routers.
- app/config.py centralizes environment-driven runtime settings, provider selection, and service endpoints.
3. Cross-cutting request handling
- Middleware adds request tracing (X-Request-ID) and HMAC authentication behavior.
- Dependencies enforce API-key access for online endpoints.
- Responses are normalized into a stable envelope with success/data/error/detail/request_id, which is important for diagramming the API boundary.
4. Router boundary: shared vs online
- app/routers/shared/* contains operational and query endpoints: health, readiness, model health, metrics, probe, classify, collections, and search.
- app/routers/online/* contains the ingestion workflow: scrape, crawl/discovery, document parse, ingest, batch ingest, streaming ingest, sparse encoding, and vector deletion.
- Utility modules under routers/_ingest_utils.py and _parse_utils.py support those pipelines.
5. Ingestion pipeline
- The core data flow is: source URL or content → scrape/parse → classify → chunk → optional contextual enrichment → embed → store in Qdrant.
- app/services/ingest/ingest_service.py orchestrates this pipeline and is the main integration point between scraping, parsing, intelligence, embeddings, and storage.
6. Scraping subsystem
- app/services/scraping/* handles web acquisition and URL discovery.
- It can use multiple upstreams: Jina Reader as the default markdown extractor, Firecrawl as fallback/alternate, and direct HTTP scraping paths when needed.
- Site crawling includes sitemap/document-discovery logic for expanding from a root site or sitemap to candidate URLs.
7. Parsing subsystem
- app/services/parsing/* converts downloaded documents into text, with parser selection based on file type or content.
- Parser implementations include PDF, DOCX, spreadsheet, text, Unstructured, and optional LlamaParse-backed parsing.
- This is the boundary where raw fetched files become normalized textual content for downstream intelligence and embedding.
8. Intelligence / enrichment subsystem
- app/services/intelligence/* contains the text-understanding stage: chunking, classification, contextual enrichment, and funding extraction.
- It uses an LLM router/client abstraction so the classifier, contextual, and funding stages can run on different providers/models.
- Prompt templates are isolated under services/intelligence/prompts, which separates domain logic from model-specific call plumbing.
9. Embedding and vector storage
- app/services/embedding/* provides dense and sparse embedding clients.
- Dense embeddings default to BGE-M3 via TEI, with an OpenAI embedding path available when explicitly requested.
- Sparse embeddings support hybrid retrieval, and qdrant_service.py handles writing and managing vectors in Qdrant.
- Stored vector metadata includes tenant/assistant/department/content-type and ACL fields, which are essential to permission-aware retrieval.
10. Search path
- app/services/search/search_service.py implements semantic and hybrid retrieval over Qdrant.
- Search is permission-aware: request context and ACL filters are applied before results are returned.
- Hybrid search combines dense and sparse retrieval and can use Reciprocal Rank Fusion.
11. Operational and governance services
- app/services/cost.py calculates stage-level usage cost from pricing.yaml.
- app/services/audit.py and app/services/metrics.py support usage logging and observability.
- app/services/cache.py and app/services/rate_limiter.py indicate Redis-backed cache/rate-limit state in the deployment.
- Usage accounting is a first-class part of ingest responses, not just an internal log.
12. External dependencies and infrastructure
- The system depends on Qdrant for vector storage, Redis for cache/rate-limit state, and optionally ClickHouse for audit/usage persistence.
- Model and content providers include OpenAI, Anthropic, Nebius/Together/Groq/Fireworks/DeepInfra for chat, TEI for embeddings, Jina Reader and Firecrawl for scraping, and LlamaParse for document parsing.
- The repo also includes production build/deploy workflows and a Dockerfile, implying containerized deployment.
13. Security and trust boundaries
- Outbound fetches are constrained by URL safety checks, and online endpoints can require both HMAC and API-key authentication.
- The architecture distinguishes between public health/readiness endpoints and authenticated ingestion/query endpoints.
- Search results are filtered by user/tenant permissions rather than exposing raw vectors.
14. How to draw the architecture diagram
- Show FastAPI as the central API layer with two router groups: shared control/query APIs and online ingestion APIs.
- Draw the ingestion flow from external content sources into scraping/parsing, then intelligence, then embeddings, then Qdrant.
- Draw the search flow from authenticated client request into permission filtering, then Qdrant retrieval, then response envelope.
- Surround the app with supporting services: Redis, ClickHouse, provider APIs, TEI, Qdrant, and observability/auth middleware.
Main entry points: main.py App main · ingest_service.py Ingest svc · api_key.py API key guard · search.py Search APIs
Components 33
FastAPI app
Application wiring, request boundary, and router registration.
Routers
Shared query endpoints and online ingestion endpoints.
- Health APIs · Health, readiness, and model/probe endpoints.
data-plane/app/routers/shared/health.py - Search APIs · Semantic and hybrid retrieval endpoints.
data-plane/app/routers/shared/search.py - Classify APIs · Classification and model-facing shared endpoints.
data-plane/app/routers/shared/classify.py - Collections API · Collection management and lookup surface.
data-plane/app/routers/shared/collections.py - Metrics API · Operational metrics and observability endpoint.
data-plane/app/routers/shared/metrics.py - Scrape API · Content acquisition, crawl, and URL discovery endpoints.
data-plane/app/routers/online/scrape.py - Parse API · Document parsing and normalization endpoints.
data-plane/app/routers/online/parse.py - Ingest API · Single-item ingestion entrypoint and orchestration surface.
data-plane/app/routers/online/ingest.py - Stream ingest · Batch and streaming ingestion variants.
data-plane/app/routers/online/ingest_stream.py - Vectors API · Sparse encoding and vector deletion operations.
data-plane/app/routers/online/vectors.py
Core pipeline
Scraping, parsing, intelligence, embedding, and search flow.
- Scraping svc · Coordinates web fetching, alternate upstreams, and discovery.
data-plane/app/services/scraping/scraper_service.py - Parsing svc · Selects and runs parsers to normalize fetched documents.
data-plane/app/services/parsing/parser_service.py - Intelligence · Chunking, classification, contextual enrichment, and funding extraction.
data-plane/app/services/intelligence - Ingest svc · Main ingestion pipeline from source content to stored vectors.
data-plane/app/services/ingest/ingest_service.py - Embedding svc · Dense and sparse embedding clients used by ingest and search.
data-plane/app/services/embedding - Search svc · Permission-aware semantic and hybrid retrieval over vectors.
data-plane/app/services/search/search_service.py
Ops & policy
Cross-cutting security, observability, cost, and stateful support services.
- Request ID · Adds and propagates X-Request-ID across requests and responses.
data-plane/app/middleware/request_id.py - HMAC auth · Authenticates signed requests for protected endpoints.
data-plane/app/middleware/hmac_auth.py - API key guard · Enforces API-key access for online routes.
data-plane/app/dependencies/api_key.py - Cache · Redis-backed cache and rate-limit state.
data-plane/app/services/cache.py - Rate limiter · Controls request throughput using shared state.
data-plane/app/services/rate_limiter.py - Audit log · Usage and audit persistence for operational accounting.
data-plane/app/services/audit.py - Costing · Stage-level usage cost calculation from pricing rules.
data-plane/app/services/cost.py
External systems
Upstream providers and infrastructure dependencies.
- Qdrant svc · Writes, manages, and retrieves vectors in Qdrant.
data-plane/app/services/embedding/qdrant_service.py - Redis · Shared cache and rate-limit backend.
- ClickHouse · Optional audit and usage persistence backend.
- TEI · Dense and sparse embedding inference service.
- OpenAI · Optional chat and embedding provider.
- LLM providers · Anthropic and other chat/model providers used by intelligence stages.
- Scrape providers · Jina Reader, Firecrawl, and related upstream acquisition services.
- LlamaParse · Optional document parsing backend.
Connections 44
- App main → Config: loads settings · App startup reads runtime configuration and provider selection.
- App main → Request ID: adds tracing · Middleware injects request IDs into the request lifecycle.
- App main → HMAC auth: protects API · Middleware enforces signed-request authentication.
- App main → Health APIs: mounts · Registers shared operational endpoints.
- App main → Search APIs: mounts · Registers shared retrieval endpoints.
- App main → Classify APIs: mounts · Registers classification endpoints.
- App main → Collections API: mounts · Registers collection endpoints.
- App main → Metrics API: mounts · Registers metrics endpoints.
- App main → Scrape API: mounts · Registers online acquisition endpoints.
- App main → Parse API: mounts · Registers online parsing endpoints.
- App main → Ingest API: mounts · Registers ingestion endpoints.
- App main → Stream ingest: mounts · Registers batch and streaming ingest endpoints.
- App main → Vectors API: mounts · Registers vector management endpoints.
- Scrape API → API key guard: requires · Online acquisition routes require API-key access.
- Parse API → API key guard: requires · Online parsing routes require API-key access.
- Ingest API → API key guard: requires · Ingest routes require API-key access.
- Stream ingest → API key guard: requires · Streaming ingest routes require API-key access.
- Vectors API → API key guard: requires · Vector operations require API-key access.
- Scrape API → Scraping svc: uses · Scrape routes delegate acquisition and discovery.
- Parse API → Parsing svc: uses · Parse routes delegate document normalization.
- Ingest API → Ingest svc: orchestrates · Ingest routes call the main pipeline orchestrator.
- Stream ingest → Ingest svc: streams to · Batch and streaming paths feed the ingest orchestrator.
- Ingest svc → Scraping svc: fetches source · Pipeline starts from acquired content or source URLs.
- Ingest svc → Parsing svc: normalizes · Pipeline turns fetched documents into text.
- Ingest svc → Intelligence: enriches · Pipeline applies classification, chunking, and contextual enrichment.
- Ingest svc → Embedding svc: embeds · Pipeline generates dense and sparse vectors.
- Ingest svc → Qdrant svc: stores · Pipeline writes vectors and metadata to Qdrant.
- Scraping svc → Scrape providers: calls · Scraping can use Jina Reader, Firecrawl, or direct fetch paths.
- Parsing svc → LlamaParse: optional · Parsing can delegate to LlamaParse when enabled.
- Parsing svc → Intelligence: passes text · Normalized text flows into downstream intelligence.
- Intelligence → LLM providers: calls · Classification, contextual, and funding stages use LLM backends.
- Intelligence → Costing: accounts for · Usage is costed by stage for ingest responses.
- Embedding svc → TEI: uses · Dense and sparse embeddings default to TEI paths.
- Embedding svc → OpenAI: fallback · OpenAI embeddings can be selected explicitly.
- Search svc → Qdrant svc: retrieves from · Search runs over stored vectors in Qdrant.
- Search APIs → Search svc: uses · Shared search routes execute retrieval service logic.
- Search svc → Request ID: returns id · Responses include request-scoped trace identifiers.
- Search svc → Cache: consults · Search can use shared state for throttling or derived data.
- Rate limiter → Redis: backed by · Rate limiting uses Redis-backed shared state.
- Cache → Redis: backed by · Cache state is stored in Redis.
- Audit log → ClickHouse: persists to · Audit and usage records can be written to ClickHouse.
- Ingest svc → Audit log: logs usage · Ingest responses include usage and audit accounting.
- Search APIs → Qdrant svc: queries · Search endpoints rely on vector retrieval.
- Search APIs → API key guard: requires · Search access is protected as a shared authenticated surface.