# ki-quadrat/knowledgebase-backend architecture

> Architecture diagram of the GitHub repository ki-quadrat/knowledgebase-backend, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ki-quadrat/knowledgebase-backend
- Repository: https://github.com/ki-quadrat/knowledgebase-backend

## Overview

1. **Overall shape**
   - This is a Python 3.13 FastAPI backend called the “data plane.”
   - It sits behind `/api/v1` and exposes two broad surfaces: **shared APIs** for health/search/classification/metrics and **online APIs** for scraping, parsing, ingesting, and vector management.

2. **Request entrypoint and app wiring**
   - `app/main.py` is the composition root: it creates the FastAPI app, wires lifespan/startup behavior, registers middleware, and mounts routers.
   - `app/config.py` centralizes environment-driven runtime settings, provider selection, and service endpoints.

3. **Cross-cutting request handling**
   - Middleware adds **request tracing** (`X-Request-ID`) and **HMAC authentication** behavior.
   - Dependencies enforce API-key access for online endpoints.
   - Responses are normalized into a stable envelope with `success/data/error/detail/request_id`, which is important for diagramming the API boundary.

4. **Router boundary: shared vs online**
   - `app/routers/shared/*` contains operational and query endpoints: health, readiness, model health, metrics, probe, classify, collections, and search.
   - `app/routers/online/*` contains the ingestion workflow: scrape, crawl/discovery, document parse, ingest, batch ingest, streaming ingest, sparse encoding, and vector deletion.
   - Utility modules under `routers/_ingest_utils.py` and `_parse_utils.py` support those pipelines.

5. **Ingestion pipeline**
   - The core data flow is: **source URL or content → scrape/parse → classify → chunk → optional contextual enrichment → embed → store in Qdrant**.
   - `app/services/ingest/ingest_service.py` orchestrates this pipeline and is the main integration point between scraping, parsing, intelligence, embeddings, and storage.

6. **Scraping subsystem**
   - `app/services/scraping/*` handles web acquisition and URL discovery.
   - It can use multiple upstreams: **Jina Reader** as the default markdown extractor, **Firecrawl** as fallback/alternate, and direct HTTP scraping paths when needed.
   - Site crawling includes sitemap/document-discovery logic for expanding from a root site or sitemap to candidate URLs.

7. **Parsing subsystem**
   - `app/services/parsing/*` converts downloaded documents into text, with parser selection based on file type or content.
   - Parser implementations include PDF, DOCX, spreadsheet, text, Unstructured, and optional LlamaParse-backed parsing.
   - This is the boundary where raw fetched files become normalized textual content for downstream intelligence and embedding.

8. **Intelligence / enrichment subsystem**
   - `app/services/intelligence/*` contains the text-understanding stage: chunking, classification, contextual enrichment, and funding extraction.
   - It uses an LLM router/client abstraction so the classifier, contextual, and funding stages can run on different providers/models.
   - Prompt templates are isolated under `services/intelligence/prompts`, which separates domain logic from model-specific call plumbing.

9. **Embedding and vector storage**
   - `app/services/embedding/*` provides dense and sparse embedding clients.
   - Dense embeddings default to **BGE-M3 via TEI**, with an OpenAI embedding path available when explicitly requested.
   - Sparse embeddings support hybrid retrieval, and `qdrant_service.py` handles writing and managing vectors in **Qdrant**.
   - Stored vector metadata includes tenant/assistant/department/content-type and ACL fields, which are essential to permission-aware retrieval.

10. **Search path**
   - `app/services/search/search_service.py` implements semantic and hybrid retrieval over Qdrant.
   - Search is permission-aware: request context and ACL filters are applied before results are returned.
   - Hybrid search combines dense and sparse retrieval and can use Reciprocal Rank Fusion.

11. **Operational and governance services**
   - `app/services/cost.py` calculates stage-level usage cost from `pricing.yaml`.
   - `app/services/audit.py` and `app/services/metrics.py` support usage logging and observability.
   - `app/services/cache.py` and `app/services/rate_limiter.py` indicate Redis-backed cache/rate-limit state in the deployment.
   - Usage accounting is a first-class part of ingest responses, not just an internal log.

12. **External dependencies and infrastructure**
   - The system depends on **Qdrant** for vector storage, **Redis** for cache/rate-limit state, and optionally **ClickHouse** for audit/usage persistence.
   - Model and content providers include **OpenAI**, **Anthropic**, **Nebius/Together/Groq/Fireworks/DeepInfra** for chat, **TEI** for embeddings, **Jina Reader** and **Firecrawl** for scraping, and **LlamaParse** for document parsing.
   - The repo also includes production build/deploy workflows and a Dockerfile, implying containerized deployment.

13. **Security and trust boundaries**
   - Outbound fetches are constrained by URL safety checks, and online endpoints can require both HMAC and API-key authentication.
   - The architecture distinguishes between public health/readiness endpoints and authenticated ingestion/query endpoints.
   - Search results are filtered by user/tenant permissions rather than exposing raw vectors.

14. **How to draw the architecture diagram**
   - Show FastAPI as the central API layer with two router groups: **shared control/query APIs** and **online ingestion APIs**.
   - Draw the ingestion flow from external content sources into scraping/parsing, then intelligence, then embeddings, then Qdrant.
   - Draw the search flow from authenticated client request into permission filtering, then Qdrant retrieval, then response envelope.
   - Surround the app with supporting services: Redis, ClickHouse, provider APIs, TEI, Qdrant, and observability/auth middleware.

Main entry points: [`data-plane/app/main.py`](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/main.py) (App main), [`data-plane/app/services/ingest/ingest_service.py`](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/services/ingest/ingest_service.py) (Ingest svc), [`data-plane/app/dependencies/api_key.py`](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/dependencies/api_key.py) (API key guard), [`data-plane/app/routers/shared/search.py`](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/routers/shared/search.py) (Search APIs)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_app["FastAPI app"]
  node_main["App main<br/>composition root<br/>[main.py]"]
  node_config["Config<br/>[config.py]"]
end

subgraph group_routers["Routers"]
  node_health["Health APIs<br/>shared router<br/>[health.py]"]
  node_shared_search["Search APIs<br/>shared router<br/>[search.py]"]
  node_classify["Classify APIs<br/>shared router<br/>[classify.py]"]
  node_shared_collections["Collections API<br/>shared router<br/>[collections.py]"]
  node_metrics["Metrics API<br/>shared router<br/>[metrics.py]"]
  node_online_scrape["Scrape API<br/>online router<br/>[scrape.py]"]
  node_online_parse["Parse API<br/>online router<br/>[parse.py]"]
  node_online_ingest["Ingest API<br/>online router<br/>[ingest.py]"]
  node_ingest_stream["Stream ingest<br/>online router<br/>[ingest_stream.py]"]
  node_online_vectors["Vectors API<br/>online router<br/>[vectors.py]"]
end

subgraph group_pipeline["Core pipeline"]
  node_scraper_service["Scraping svc<br/>acquisition<br/>[scraper_service.py]"]
  node_parser_service["Parsing svc<br/>document parsing<br/>[parser_service.py]"]
  node_intelligence{{"Intelligence<br/>enrichment"}}
  node_ingest_service["Ingest svc<br/>pipeline orchestrator<br/>[ingest_service.py]"]
  node_embedding["Embedding svc<br/>embeddings"]
  node_search_service["Search svc<br/>retrieval<br/>[search_service.py]"]
end

subgraph group_ops["Ops & policy"]
  node_request_id["Request ID<br/>middleware<br/>[request_id.py]"]
  node_hmac_auth["HMAC auth<br/>middleware<br/>[hmac_auth.py]"]
  node_api_key["API key guard<br/>dependency<br/>[api_key.py]"]
  node_cache[("Cache<br/>redis state<br/>[cache.py]")]
  node_rate_limiter["Rate limiter<br/>policy<br/>[rate_limiter.py]"]
  node_audit["Audit log<br/>usage logging<br/>[audit.py]"]
  node_cost["Costing<br/>usage accounting<br/>[cost.py]"]
end

subgraph group_external["External systems"]
  node_qdrant_service[("Qdrant svc<br/>vector store<br/>[qdrant_service.py]")]
  node_redis[("Redis<br/>external state")]
  node_clickhouse[("ClickHouse<br/>external audit store")]
  node_tei["TEI<br/>embedding api"]
  node_openai["OpenAI<br/>model api"]
  node_llm_providers["LLM providers<br/>model backends"]
  node_jina_firecrawl["Scrape providers<br/>content fetchers"]
  node_llama_parse["LlamaParse<br/>document parser api"]
end

node_main -->|"loads settings"| node_config
node_main -->|"adds tracing"| node_request_id
node_main -->|"protects API"| node_hmac_auth
node_main -->|"mounts"| node_health
node_main -->|"mounts"| node_shared_search
node_main -->|"mounts"| node_classify
node_main -->|"mounts"| node_shared_collections
node_main -->|"mounts"| node_metrics
node_main -->|"mounts"| node_online_scrape
node_main -->|"mounts"| node_online_parse
node_main -->|"mounts"| node_online_ingest
node_main -->|"mounts"| node_ingest_stream
node_main -->|"mounts"| node_online_vectors
node_online_scrape -->|"requires"| node_api_key
node_online_parse -->|"requires"| node_api_key
node_online_ingest -->|"requires"| node_api_key
node_ingest_stream -->|"requires"| node_api_key
node_online_vectors -->|"requires"| node_api_key
node_online_scrape -->|"uses"| node_scraper_service
node_online_parse -->|"uses"| node_parser_service
node_online_ingest -->|"orchestrates"| node_ingest_service
node_ingest_stream -->|"streams to"| node_ingest_service
node_ingest_service -->|"fetches source"| node_scraper_service
node_ingest_service -->|"normalizes"| node_parser_service
node_ingest_service -->|"enriches"| node_intelligence
node_ingest_service -->|"embeds"| node_embedding
node_ingest_service -->|"stores"| node_qdrant_service
node_scraper_service -->|"calls"| node_jina_firecrawl
node_parser_service -.->|"optional"| node_llama_parse
node_parser_service -->|"passes text"| node_intelligence
node_intelligence -->|"calls"| node_llm_providers
node_intelligence -->|"accounts for"| node_cost
node_embedding -->|"uses"| node_tei
node_embedding -.->|"fallback"| node_openai
node_search_service -->|"retrieves from"| node_qdrant_service
node_shared_search -->|"uses"| node_search_service
node_search_service -->|"returns id"| node_request_id
node_search_service -.->|"consults"| node_cache
node_rate_limiter -->|"backed by"| node_redis
node_cache -->|"backed by"| node_redis
node_audit -.->|"persists to"| node_clickhouse
node_ingest_service -->|"logs usage"| node_audit
node_shared_search -->|"queries"| node_qdrant_service
node_shared_search -->|"requires"| node_api_key

click node_main "https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/main.py"
click node_config "https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/config.py"
click node_request_id "https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/middleware/request_id.py"
click node_hmac_auth "https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/middleware/hmac_auth.py"
click node_api_key "https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/dependencies/api_key.py"
click node_health "https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/routers/shared/health.py"
click node_shared_search "https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/routers/shared/search.py"
click node_classify "https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/routers/shared/classify.py"
click node_shared_collections "https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/routers/shared/collections.py"
click node_metrics "https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/routers/shared/metrics.py"
click node_online_scrape "https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/routers/online/scrape.py"
click node_online_parse "https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/routers/online/parse.py"
click node_online_ingest "https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/routers/online/ingest.py"
click node_ingest_stream "https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/routers/online/ingest_stream.py"
click node_online_vectors "https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/routers/online/vectors.py"
click node_scraper_service "https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/services/scraping/scraper_service.py"
click node_parser_service "https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/services/parsing/parser_service.py"
click node_intelligence "https://github.com/ki-quadrat/knowledgebase-backend/tree/monolith-dev/data-plane/app/services/intelligence"
click node_ingest_service "https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/services/ingest/ingest_service.py"
click node_embedding "https://github.com/ki-quadrat/knowledgebase-backend/tree/monolith-dev/data-plane/app/services/embedding"
click node_qdrant_service "https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/services/embedding/qdrant_service.py"
click node_search_service "https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/services/search/search_service.py"
click node_cache "https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/services/cache.py"
click node_rate_limiter "https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/services/rate_limiter.py"
click node_audit "https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/services/audit.py"
click node_cost "https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/services/cost.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_config toneBlue
class node_health,node_shared_search,node_classify,node_shared_collections,node_metrics,node_online_scrape,node_online_parse,node_online_ingest,node_ingest_stream,node_online_vectors toneAmber
class node_scraper_service,node_parser_service,node_intelligence,node_ingest_service,node_embedding,node_search_service toneMint
class node_request_id,node_hmac_auth,node_api_key,node_cache,node_rate_limiter,node_audit,node_cost toneRose
class node_qdrant_service,node_redis,node_clickhouse,node_tei,node_openai,node_llm_providers,node_jina_firecrawl,node_llama_parse toneIndigo
```

## Components

### FastAPI app

Application wiring, request boundary, and router registration.

- [App main](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/main.py): composition root · Creates the FastAPI app, wires startup/lifespan, middleware, and routers. · `data-plane/app/main.py`
- [Config](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/config.py): runtime config · Environment-driven settings, provider selection, and service endpoints. · `data-plane/app/config.py`

### Routers

Shared query endpoints and online ingestion endpoints.

- [Health APIs](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/routers/shared/health.py): shared router · Health, readiness, and model/probe endpoints. · `data-plane/app/routers/shared/health.py`
- [Search APIs](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/routers/shared/search.py): shared router · Semantic and hybrid retrieval endpoints. · `data-plane/app/routers/shared/search.py`
- [Classify APIs](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/routers/shared/classify.py): shared router · Classification and model-facing shared endpoints. · `data-plane/app/routers/shared/classify.py`
- [Collections API](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/routers/shared/collections.py): shared router · Collection management and lookup surface. · `data-plane/app/routers/shared/collections.py`
- [Metrics API](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/routers/shared/metrics.py): shared router · Operational metrics and observability endpoint. · `data-plane/app/routers/shared/metrics.py`
- [Scrape API](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/routers/online/scrape.py): online router · Content acquisition, crawl, and URL discovery endpoints. · `data-plane/app/routers/online/scrape.py`
- [Parse API](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/routers/online/parse.py): online router · Document parsing and normalization endpoints. · `data-plane/app/routers/online/parse.py`
- [Ingest API](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/routers/online/ingest.py): online router · Single-item ingestion entrypoint and orchestration surface. · `data-plane/app/routers/online/ingest.py`
- [Stream ingest](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/routers/online/ingest_stream.py): online router · Batch and streaming ingestion variants. · `data-plane/app/routers/online/ingest_stream.py`
- [Vectors API](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/routers/online/vectors.py): online router · Sparse encoding and vector deletion operations. · `data-plane/app/routers/online/vectors.py`

### Core pipeline

Scraping, parsing, intelligence, embedding, and search flow.

- [Scraping svc](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/services/scraping/scraper_service.py): acquisition · Coordinates web fetching, alternate upstreams, and discovery. · `data-plane/app/services/scraping/scraper_service.py`
- [Parsing svc](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/services/parsing/parser_service.py): document parsing · Selects and runs parsers to normalize fetched documents. · `data-plane/app/services/parsing/parser_service.py`
- [Intelligence](https://github.com/ki-quadrat/knowledgebase-backend/tree/monolith-dev/data-plane/app/services/intelligence): enrichment · Chunking, classification, contextual enrichment, and funding extraction. · `data-plane/app/services/intelligence`
- [Ingest svc](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/services/ingest/ingest_service.py): pipeline orchestrator · Main ingestion pipeline from source content to stored vectors. · `data-plane/app/services/ingest/ingest_service.py`
- [Embedding svc](https://github.com/ki-quadrat/knowledgebase-backend/tree/monolith-dev/data-plane/app/services/embedding): embeddings · Dense and sparse embedding clients used by ingest and search. · `data-plane/app/services/embedding`
- [Search svc](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/services/search/search_service.py): retrieval · Permission-aware semantic and hybrid retrieval over vectors. · `data-plane/app/services/search/search_service.py`

### Ops & policy

Cross-cutting security, observability, cost, and stateful support services.

- [Request ID](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/middleware/request_id.py): middleware · Adds and propagates X-Request-ID across requests and responses. · `data-plane/app/middleware/request_id.py`
- [HMAC auth](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/middleware/hmac_auth.py): middleware · Authenticates signed requests for protected endpoints. · `data-plane/app/middleware/hmac_auth.py`
- [API key guard](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/dependencies/api_key.py): dependency · Enforces API-key access for online routes. · `data-plane/app/dependencies/api_key.py`
- [Cache](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/services/cache.py): redis state · Redis-backed cache and rate-limit state. · `data-plane/app/services/cache.py`
- [Rate limiter](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/services/rate_limiter.py): policy · Controls request throughput using shared state. · `data-plane/app/services/rate_limiter.py`
- [Audit log](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/services/audit.py): usage logging · Usage and audit persistence for operational accounting. · `data-plane/app/services/audit.py`
- [Costing](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/services/cost.py): usage accounting · Stage-level usage cost calculation from pricing rules. · `data-plane/app/services/cost.py`

### External systems

Upstream providers and infrastructure dependencies.

- [Qdrant svc](https://github.com/ki-quadrat/knowledgebase-backend/blob/monolith-dev/data-plane/app/services/embedding/qdrant_service.py): vector store · Writes, manages, and retrieves vectors in Qdrant. · `data-plane/app/services/embedding/qdrant_service.py`
- **Redis**: external state · Shared cache and rate-limit backend.
- **ClickHouse**: external audit store · Optional audit and usage persistence backend.
- **TEI**: embedding api · Dense and sparse embedding inference service.
- **OpenAI**: model api · Optional chat and embedding provider.
- **LLM providers**: model backends · Anthropic and other chat/model providers used by intelligence stages.
- **Scrape providers**: content fetchers · Jina Reader, Firecrawl, and related upstream acquisition services.
- **LlamaParse**: document parser api · Optional document parsing backend.

## Connections

- App main → Config: loads settings · App startup reads runtime configuration and provider selection.
- App main → Request ID: adds tracing · Middleware injects request IDs into the request lifecycle.
- App main → HMAC auth: protects API · Middleware enforces signed-request authentication.
- App main → Health APIs: mounts · Registers shared operational endpoints.
- App main → Search APIs: mounts · Registers shared retrieval endpoints.
- App main → Classify APIs: mounts · Registers classification endpoints.
- App main → Collections API: mounts · Registers collection endpoints.
- App main → Metrics API: mounts · Registers metrics endpoints.
- App main → Scrape API: mounts · Registers online acquisition endpoints.
- App main → Parse API: mounts · Registers online parsing endpoints.
- App main → Ingest API: mounts · Registers ingestion endpoints.
- App main → Stream ingest: mounts · Registers batch and streaming ingest endpoints.
- App main → Vectors API: mounts · Registers vector management endpoints.
- Scrape API → API key guard: requires · Online acquisition routes require API-key access.
- Parse API → API key guard: requires · Online parsing routes require API-key access.
- Ingest API → API key guard: requires · Ingest routes require API-key access.
- Stream ingest → API key guard: requires · Streaming ingest routes require API-key access.
- Vectors API → API key guard: requires · Vector operations require API-key access.
- Scrape API → Scraping svc: uses · Scrape routes delegate acquisition and discovery.
- Parse API → Parsing svc: uses · Parse routes delegate document normalization.
- Ingest API → Ingest svc: orchestrates · Ingest routes call the main pipeline orchestrator.
- Stream ingest → Ingest svc: streams to · Batch and streaming paths feed the ingest orchestrator.
- Ingest svc → Scraping svc: fetches source · Pipeline starts from acquired content or source URLs.
- Ingest svc → Parsing svc: normalizes · Pipeline turns fetched documents into text.
- Ingest svc → Intelligence: enriches · Pipeline applies classification, chunking, and contextual enrichment.
- Ingest svc → Embedding svc: embeds · Pipeline generates dense and sparse vectors.
- Ingest svc → Qdrant svc: stores · Pipeline writes vectors and metadata to Qdrant.
- Scraping svc → Scrape providers: calls · Scraping can use Jina Reader, Firecrawl, or direct fetch paths.
- Parsing svc → LlamaParse: optional · Parsing can delegate to LlamaParse when enabled.
- Parsing svc → Intelligence: passes text · Normalized text flows into downstream intelligence.
- Intelligence → LLM providers: calls · Classification, contextual, and funding stages use LLM backends.
- Intelligence → Costing: accounts for · Usage is costed by stage for ingest responses.
- Embedding svc → TEI: uses · Dense and sparse embeddings default to TEI paths.
- Embedding svc → OpenAI: fallback · OpenAI embeddings can be selected explicitly.
- Search svc → Qdrant svc: retrieves from · Search runs over stored vectors in Qdrant.
- Search APIs → Search svc: uses · Shared search routes execute retrieval service logic.
- Search svc → Request ID: returns id · Responses include request-scoped trace identifiers.
- Search svc → Cache: consults · Search can use shared state for throttling or derived data.
- Rate limiter → Redis: backed by · Rate limiting uses Redis-backed shared state.
- Cache → Redis: backed by · Cache state is stored in Redis.
- Audit log → ClickHouse: persists to · Audit and usage records can be written to ClickHouse.
- Ingest svc → Audit log: logs usage · Ingest responses include usage and audit accounting.
- Search APIs → Qdrant svc: queries · Search endpoints rely on vector retrieval.
- Search APIs → API key guard: requires · Search access is protected as a shared authenticated surface.