How servercrash358/freshdex works
Freshdex keeps a Postgres-backed RAG index synchronized with document changes: writes emit Debezium CDC events, which are transformed and split for embedding or deletion, then applied to pgvector. Users query through a retrieval, reranking, and generation API with a Redis answer cache; a separate CDC consumer invalidates answers citing changed documents. Evaluation measures indexing freshness and answer quality. The README documents the broker and monitoring stack; source excerpts show service internals, while several documented integrations have no sampled implementation evidence.
Main entry points: query.py Query endpoint · docker-compose.yml Redpanda topics · consumer.py Checksum and routing · freshness_benchmark.py Freshness benchmark
Components 24
CDC and indexing
- Postgres documents
db/init/01_schema.sql - Debezium CDC
connectors/debezium-postgres-connector.json - Redpanda topics
docker-compose.yml - Checksum and routing
services/transform-layer/app/consumer.py - Checksum store
services/transform-layer/app/checksum_store.py - Embedding requests
services/transform-layer/app/config.py - Deletion events
services/transform-layer/app/config.py - Request consumer
services/embedding-worker/app/consumer.py - Pluggable embedder
services/embedding-worker/app/embedder.py - Tombstone consumer
services/embedding-worker/app/tombstone_consumer.py - Indexed documents
db/init/02_indexed_documents.sql
Query and cache
- Cache invalidator
services/cache-invalidator/app/consumer.py - Redis answer cache
docker-compose.yml - Query endpoint
services/query-api/app/routers/query.py - Vector retriever
services/query-api/app/retriever.py - Reranker
services/query-api/app/reranker.py - Answer generator
services/query-api/app/generator.py
Evaluation and observability
- Scheduled evaluations
evals/flow.py - Freshness benchmark
evals/freshness_benchmark.py - RAGAS scoring
evals/ragas_eval.py - Pushgateway
docker-compose.yml - Prometheus and Grafana
observability/prometheus.yml
Other components
- Questioner
- Document writer
Connections 29
- Document writer → Postgres documents: writes documents
README.md - Postgres documents → Debezium CDC: publishes CDC
README.md - Debezium CDC → Redpanda topics: routes events
README.md - Redpanda topics → Checksum and routing: consumes outbox
services/transform-layer/app/consumer.py - Checksum and routing → Checksum store: checks checksum
services/transform-layer/app/consumer.py - Checksum and routing → Embedding requests: publishes changes
services/transform-layer/app/consumer.py - Checksum and routing → Deletion events: routes deletions
services/transform-layer/app/consumer.py - Redpanda topics → Request consumer: consumes requests
README.md - Request consumer → Pluggable embedder: embeds content
services/embedding-worker/app/consumer.py - Request consumer → Indexed documents: upserts vectors
- Redpanda topics → Tombstone consumer: consumes deletions
README.md - Tombstone consumer → Indexed documents: marks deleted
services/embedding-worker/app/tombstone_consumer.py - Redpanda topics → Cache invalidator: consumes raw CDC
services/cache-invalidator/app/consumer.py - Cache invalidator → Redis answer cache: evicts cited answers
services/cache-invalidator/app/consumer.py - Questioner → Query endpoint: submits query
README.md - Query endpoint → Redis answer cache: checks and caches
services/query-api/app/routers/query.py - Query endpoint → Vector retriever: searches candidates
services/query-api/app/routers/query.py - Vector retriever → Indexed documents: searches vectors
services/query-api/app/retriever.py - Query endpoint → Reranker: reranks results
services/query-api/app/routers/query.py - Query endpoint → Answer generator: generates answer
services/query-api/app/routers/query.py - Scheduled evaluations → Freshness benchmark: runs benchmark
evals/flow.py - Scheduled evaluations → RAGAS scoring: runs quality eval
evals/flow.py - Freshness benchmark → Postgres documents: writes canary
evals/freshness_benchmark.py - Freshness benchmark → Query endpoint: polls retrieval
evals/freshness_benchmark.py - RAGAS scoring → Postgres documents: seeds corpus
evals/ragas_eval.py - RAGAS scoring → Query endpoint: queries answers
evals/ragas_eval.py - Freshness benchmark → Pushgateway: pushes latency
evals/freshness_benchmark.py - RAGAS scoring → Pushgateway: pushes scores
evals/ragas_eval.py - Prometheus and Grafana → Pushgateway: scrapes metrics
README.md