# servercrash358/freshdex architecture

> Architecture diagram of the GitHub repository servercrash358/freshdex, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/servercrash358/freshdex
- Repository: https://github.com/servercrash358/freshdex

## Overview

Freshdex keeps a Postgres-backed RAG index synchronized with document changes: writes emit Debezium CDC events, which are transformed and split for embedding or deletion, then applied to pgvector. Users query through a retrieval, reranking, and generation API with a Redis answer cache; a separate CDC consumer invalidates answers citing changed documents. Evaluation measures indexing freshness and answer quality. The README documents the broker and monitoring stack; source excerpts show service internals, while several documented integrations have no sampled implementation evidence.

Main entry points: [`services/query-api/app/routers/query.py`](https://github.com/servercrash358/freshdex/blob/main/services/query-api/app/routers/query.py) (Query endpoint), [`docker-compose.yml`](https://github.com/servercrash358/freshdex/blob/main/docker-compose.yml) (Redpanda topics), [`services/transform-layer/app/consumer.py`](https://github.com/servercrash358/freshdex/blob/main/services/transform-layer/app/consumer.py) (Checksum and routing), [`evals/freshness_benchmark.py`](https://github.com/servercrash358/freshdex/blob/main/evals/freshness_benchmark.py) (Freshness benchmark)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["CDC and indexing"]
  node_postgres[("Postgres documents<br/>[01_schema.sql]")]
  node_debezium["Debezium CDC"]
  node_broker["Redpanda topics<br/>[docker-compose.yml]"]
  node_transform["Checksum and routing<br/>[consumer.py]"]
  node_checksum[("Checksum store<br/>[checksum_store.py]")]
  node_requests["Embedding requests<br/>[config.py]"]
  node_tombstones["Deletion events<br/>[config.py]"]
  node_embed_consumer["Request consumer<br/>[consumer.py]"]
  node_embedder["Pluggable embedder<br/>[embedder.py]"]
  node_tomb_consumer["Tombstone consumer"]
  node_index_db[("Indexed documents")]
end

subgraph group_query["Query and cache"]
  node_cache_invalidator["Cache invalidator<br/>[consumer.py]"]
  node_redis[("Redis answer cache<br/>[docker-compose.yml]")]
  node_query_api["Query endpoint<br/>[query.py]"]
  node_retriever["Vector retriever<br/>[retriever.py]"]
  node_reranker["Reranker<br/>[reranker.py]"]
  node_generator["Answer generator<br/>[generator.py]"]
end

subgraph group_eval["Evaluation and observability"]
  node_eval_flow["Scheduled evaluations<br/>[flow.py]"]
  node_freshness["Freshness benchmark"]
  node_ragas["RAGAS scoring<br/>[ragas_eval.py]"]
  node_pushgateway["Pushgateway<br/>[docker-compose.yml]"]
  node_monitoring["Prometheus and Grafana<br/>[prometheus.yml]"]
end

node_user(("Questioner"))
node_source(("Document writer"))

node_source -->|"writes documents"| node_postgres
node_postgres -->|"publishes CDC"| node_debezium
node_debezium -->|"routes events"| node_broker
node_broker -->|"consumes outbox"| node_transform
node_transform -->|"checks checksum"| node_checksum
node_transform -->|"publishes changes"| node_requests
node_transform -->|"routes deletions"| node_tombstones
node_broker -->|"consumes requests"| node_embed_consumer
node_embed_consumer -->|"embeds content"| node_embedder
node_embed_consumer -->|"upserts vectors"| node_index_db
node_broker -->|"consumes deletions"| node_tomb_consumer
node_tomb_consumer -->|"marks deleted"| node_index_db
node_broker -->|"consumes raw CDC"| node_cache_invalidator
node_cache_invalidator -->|"evicts cited answers"| node_redis
node_user -->|"submits query"| node_query_api
node_query_api -->|"checks and caches"| node_redis
node_query_api -->|"searches candidates"| node_retriever
node_retriever -->|"searches vectors"| node_index_db
node_query_api -->|"reranks results"| node_reranker
node_query_api -->|"generates answer"| node_generator
node_eval_flow -->|"runs benchmark"| node_freshness
node_eval_flow -->|"runs quality eval"| node_ragas
node_freshness -->|"writes canary"| node_postgres
node_freshness -->|"polls retrieval"| node_query_api
node_ragas -->|"seeds corpus"| node_postgres
node_ragas -->|"queries answers"| node_query_api
node_freshness -->|"pushes latency"| node_pushgateway
node_ragas -->|"pushes scores"| node_pushgateway
node_monitoring -.->|"scrapes metrics"| node_pushgateway

click node_postgres "https://github.com/servercrash358/freshdex/blob/main/db/init/01_schema.sql"
click node_debezium "https://github.com/servercrash358/freshdex/blob/main/connectors/debezium-postgres-connector.json"
click node_broker "https://github.com/servercrash358/freshdex/blob/main/docker-compose.yml"
click node_transform "https://github.com/servercrash358/freshdex/blob/main/services/transform-layer/app/consumer.py"
click node_checksum "https://github.com/servercrash358/freshdex/blob/main/services/transform-layer/app/checksum_store.py"
click node_requests "https://github.com/servercrash358/freshdex/blob/main/services/transform-layer/app/config.py"
click node_tombstones "https://github.com/servercrash358/freshdex/blob/main/services/transform-layer/app/config.py"
click node_embed_consumer "https://github.com/servercrash358/freshdex/blob/main/services/embedding-worker/app/consumer.py"
click node_embedder "https://github.com/servercrash358/freshdex/blob/main/services/embedding-worker/app/embedder.py"
click node_tomb_consumer "https://github.com/servercrash358/freshdex/blob/main/services/embedding-worker/app/tombstone_consumer.py"
click node_index_db "https://github.com/servercrash358/freshdex/blob/main/db/init/02_indexed_documents.sql"
click node_cache_invalidator "https://github.com/servercrash358/freshdex/blob/main/services/cache-invalidator/app/consumer.py"
click node_redis "https://github.com/servercrash358/freshdex/blob/main/docker-compose.yml"
click node_query_api "https://github.com/servercrash358/freshdex/blob/main/services/query-api/app/routers/query.py"
click node_retriever "https://github.com/servercrash358/freshdex/blob/main/services/query-api/app/retriever.py"
click node_reranker "https://github.com/servercrash358/freshdex/blob/main/services/query-api/app/reranker.py"
click node_generator "https://github.com/servercrash358/freshdex/blob/main/services/query-api/app/generator.py"
click node_eval_flow "https://github.com/servercrash358/freshdex/blob/main/evals/flow.py"
click node_freshness "https://github.com/servercrash358/freshdex/blob/main/evals/freshness_benchmark.py"
click node_ragas "https://github.com/servercrash358/freshdex/blob/main/evals/ragas_eval.py"
click node_pushgateway "https://github.com/servercrash358/freshdex/blob/main/docker-compose.yml"
click node_monitoring "https://github.com/servercrash358/freshdex/blob/main/observability/prometheus.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_postgres,node_debezium,node_broker,node_transform,node_checksum,node_requests,node_tombstones,node_embed_consumer,node_embedder,node_tomb_consumer,node_index_db toneBlue
class node_cache_invalidator,node_redis,node_query_api,node_retriever,node_reranker,node_generator toneAmber
class node_eval_flow,node_freshness,node_ragas,node_pushgateway,node_monitoring toneMint
class node_user,node_source toneIndigo
```

## Components

### CDC and indexing

- [Postgres documents](https://github.com/servercrash358/freshdex/blob/main/db/init/01_schema.sql): `db/init/01_schema.sql`
- [Debezium CDC](https://github.com/servercrash358/freshdex/blob/main/connectors/debezium-postgres-connector.json): `connectors/debezium-postgres-connector.json`
- [Redpanda topics](https://github.com/servercrash358/freshdex/blob/main/docker-compose.yml): `docker-compose.yml`
- [Checksum and routing](https://github.com/servercrash358/freshdex/blob/main/services/transform-layer/app/consumer.py): `services/transform-layer/app/consumer.py`
- [Checksum store](https://github.com/servercrash358/freshdex/blob/main/services/transform-layer/app/checksum_store.py): `services/transform-layer/app/checksum_store.py`
- [Embedding requests](https://github.com/servercrash358/freshdex/blob/main/services/transform-layer/app/config.py): `services/transform-layer/app/config.py`
- [Deletion events](https://github.com/servercrash358/freshdex/blob/main/services/transform-layer/app/config.py): `services/transform-layer/app/config.py`
- [Request consumer](https://github.com/servercrash358/freshdex/blob/main/services/embedding-worker/app/consumer.py): `services/embedding-worker/app/consumer.py`
- [Pluggable embedder](https://github.com/servercrash358/freshdex/blob/main/services/embedding-worker/app/embedder.py): `services/embedding-worker/app/embedder.py`
- [Tombstone consumer](https://github.com/servercrash358/freshdex/blob/main/services/embedding-worker/app/tombstone_consumer.py): `services/embedding-worker/app/tombstone_consumer.py`
- [Indexed documents](https://github.com/servercrash358/freshdex/blob/main/db/init/02_indexed_documents.sql): `db/init/02_indexed_documents.sql`

### Query and cache

- [Cache invalidator](https://github.com/servercrash358/freshdex/blob/main/services/cache-invalidator/app/consumer.py): `services/cache-invalidator/app/consumer.py`
- [Redis answer cache](https://github.com/servercrash358/freshdex/blob/main/docker-compose.yml): `docker-compose.yml`
- [Query endpoint](https://github.com/servercrash358/freshdex/blob/main/services/query-api/app/routers/query.py): `services/query-api/app/routers/query.py`
- [Vector retriever](https://github.com/servercrash358/freshdex/blob/main/services/query-api/app/retriever.py): `services/query-api/app/retriever.py`
- [Reranker](https://github.com/servercrash358/freshdex/blob/main/services/query-api/app/reranker.py): `services/query-api/app/reranker.py`
- [Answer generator](https://github.com/servercrash358/freshdex/blob/main/services/query-api/app/generator.py): `services/query-api/app/generator.py`

### Evaluation and observability

- [Scheduled evaluations](https://github.com/servercrash358/freshdex/blob/main/evals/flow.py): `evals/flow.py`
- [Freshness benchmark](https://github.com/servercrash358/freshdex/blob/main/evals/freshness_benchmark.py): `evals/freshness_benchmark.py`
- [RAGAS scoring](https://github.com/servercrash358/freshdex/blob/main/evals/ragas_eval.py): `evals/ragas_eval.py`
- [Pushgateway](https://github.com/servercrash358/freshdex/blob/main/docker-compose.yml): `docker-compose.yml`
- [Prometheus and Grafana](https://github.com/servercrash358/freshdex/blob/main/observability/prometheus.yml): `observability/prometheus.yml`

### Other components

- **Questioner**
- **Document writer**

## Connections

- Document writer → Postgres documents: writes documents (evidence: `README.md`)
- Postgres documents → Debezium CDC: publishes CDC (evidence: `README.md`)
- Debezium CDC → Redpanda topics: routes events (evidence: `README.md`)
- Redpanda topics → Checksum and routing: consumes outbox (evidence: `services/transform-layer/app/consumer.py`)
- Checksum and routing → Checksum store: checks checksum (evidence: `services/transform-layer/app/consumer.py`)
- Checksum and routing → Embedding requests: publishes changes (evidence: `services/transform-layer/app/consumer.py`)
- Checksum and routing → Deletion events: routes deletions (evidence: `services/transform-layer/app/consumer.py`)
- Redpanda topics → Request consumer: consumes requests (evidence: `README.md`)
- Request consumer → Pluggable embedder: embeds content (evidence: `services/embedding-worker/app/consumer.py`)
- Request consumer → Indexed documents: upserts vectors
- Redpanda topics → Tombstone consumer: consumes deletions (evidence: `README.md`)
- Tombstone consumer → Indexed documents: marks deleted (evidence: `services/embedding-worker/app/tombstone_consumer.py`)
- Redpanda topics → Cache invalidator: consumes raw CDC (evidence: `services/cache-invalidator/app/consumer.py`)
- Cache invalidator → Redis answer cache: evicts cited answers (evidence: `services/cache-invalidator/app/consumer.py`)
- Questioner → Query endpoint: submits query (evidence: `README.md`)
- Query endpoint → Redis answer cache: checks and caches (evidence: `services/query-api/app/routers/query.py`)
- Query endpoint → Vector retriever: searches candidates (evidence: `services/query-api/app/routers/query.py`)
- Vector retriever → Indexed documents: searches vectors (evidence: `services/query-api/app/retriever.py`)
- Query endpoint → Reranker: reranks results (evidence: `services/query-api/app/routers/query.py`)
- Query endpoint → Answer generator: generates answer (evidence: `services/query-api/app/routers/query.py`)
- Scheduled evaluations → Freshness benchmark: runs benchmark (evidence: `evals/flow.py`)
- Scheduled evaluations → RAGAS scoring: runs quality eval (evidence: `evals/flow.py`)
- Freshness benchmark → Postgres documents: writes canary (evidence: `evals/freshness_benchmark.py`)
- Freshness benchmark → Query endpoint: polls retrieval (evidence: `evals/freshness_benchmark.py`)
- RAGAS scoring → Postgres documents: seeds corpus (evidence: `evals/ragas_eval.py`)
- RAGAS scoring → Query endpoint: queries answers (evidence: `evals/ragas_eval.py`)
- Freshness benchmark → Pushgateway: pushes latency (evidence: `evals/freshness_benchmark.py`)
- RAGAS scoring → Pushgateway: pushes scores (evidence: `evals/ragas_eval.py`)
- Prometheus and Grafana → Pushgateway: scrapes metrics (evidence: `README.md`)
