# soummo19/prismsearch architecture

> Architecture diagram of the GitHub repository soummo19/prismsearch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/soummo19/prismsearch
- Repository: https://github.com/soummo19/prismsearch

## Overview

This is a near-real-time, multi-source retrieval and grounded RAG system. Its principal workflow captures PostgreSQL changes through Debezium and Redpanda, normalizes them into canonical documents, indexes text and vectors, then serves tenant-scoped hybrid search and cited answers. The graph also shows the ML service, semantic and response caching, and storage layers. Some internal query and indexing calls are omitted from the excerpts; documented relationships are cited to the README, and uncertain links are not asserted.

Main entry points: [`cmd/query/main.go`](https://github.com/soummo19/prismsearch/blob/main/cmd/query/main.go) (Query API), [`cmd/indexer/main.go`](https://github.com/soummo19/prismsearch/blob/main/cmd/indexer/main.go) (Indexer service), [`cmd/normalizer/main.go`](https://github.com/soummo19/prismsearch/blob/main/cmd/normalizer/main.go) (Normalizer service), [`internal/rag/rag.go`](https://github.com/soummo19/prismsearch/blob/main/internal/rag/rag.go) (Cited RAG)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Change ingestion"]
  node_redpanda["Redpanda"]
  node_normalizer_cmd["Normalizer service<br/>[main.go]"]
  node_normalizer["CDC normalizer<br/>[normalizer.go]"]
  node_canonical["Canonical document<br/>[doc.go]"]
  node_priority_topics["Fast / bulk topics"]
  node_indexer["Indexer service<br/>[main.go]"]
  node_idempotency["Version guard<br/>[guard.go]"]
end

subgraph group_search["Retrieval and RAG"]
  node_query["Query API<br/>[main.go]"]
  node_fusion["Hybrid fusion<br/>[fusion.go]"]
  node_rag["Cited RAG<br/>[rag.go]"]
  node_faithfulness["Faithfulness guard<br/>[faithfulness.go]"]
  node_semantic_cache["Semantic cache<br/>[semcache.go]"]
  node_response_cache["Response cache<br/>[cache.go]"]
  node_answer["Cited answer"]
end

subgraph group_ml["ML inference"]
  node_ml_client["ML client<br/>[mlclient.go]"]
  node_ml_api["ML HTTP service<br/>[app.py]"]
  node_embedder["Embedding backend<br/>[embedder.py]"]
  node_reranker["Reranker backend<br/>[reranker.py]"]
end

subgraph group_infra["Stores and telemetry"]
  node_opensearch[("OpenSearch BM25<br/>[opensearch.go]")]
  node_qdrant[("Qdrant vectors<br/>[qdrant.go]")]
  node_kafka_client["Kafka client<br/>[kafka.go]"]
  node_metrics["Metrics<br/>[metrics.go]"]
end

node_postgres[("PostgreSQL source")]
node_debezium["Debezium CDC"]
node_query_user(("User"))

node_postgres -->|"streams changes"| node_debezium
node_debezium -->|"publishes CDC"| node_redpanda
node_redpanda -->|"feeds events"| node_normalizer_cmd
node_normalizer_cmd -->|"parses events"| node_normalizer
node_normalizer -->|"creates documents"| node_canonical
node_normalizer_cmd -->|"routes by tier"| node_priority_topics
node_priority_topics -->|"feeds lanes"| node_indexer
node_indexer -->|"checks versions"| node_idempotency
node_indexer -->|"requests embeddings"| node_ml_client
node_indexer -->|"writes documents"| node_opensearch
node_indexer -->|"writes vectors"| node_qdrant
node_query_user -->|"submits search"| node_query
node_query -->|"runs BM25"| node_opensearch
node_query -->|"runs vector search"| node_qdrant
node_query -->|"combines rankings"| node_fusion
node_query -->|"requests reranking"| node_ml_client
node_query -->|"checks semantic match"| node_semantic_cache
node_query -->|"checks cached response"| node_response_cache
node_query -->|"generates answer"| node_rag
node_rag -->|"checks grounding"| node_faithfulness
node_rag -->|"returns citations"| node_answer
node_answer -->|"returns answer"| node_query_user
node_ml_client -->|"calls HTTP endpoints"| node_ml_api
node_ml_api -->|"embeds texts"| node_embedder
node_ml_api -->|"scores documents"| node_reranker
node_indexer -->|"records freshness"| node_metrics

click node_normalizer_cmd "https://github.com/soummo19/prismsearch/blob/main/cmd/normalizer/main.go"
click node_normalizer "https://github.com/soummo19/prismsearch/blob/main/internal/normalizer/normalizer.go"
click node_canonical "https://github.com/soummo19/prismsearch/blob/main/internal/contract/doc.go"
click node_indexer "https://github.com/soummo19/prismsearch/blob/main/cmd/indexer/main.go"
click node_idempotency "https://github.com/soummo19/prismsearch/blob/main/internal/idempotency/guard.go"
click node_query "https://github.com/soummo19/prismsearch/blob/main/cmd/query/main.go"
click node_fusion "https://github.com/soummo19/prismsearch/blob/main/internal/retrieve/fusion.go"
click node_rag "https://github.com/soummo19/prismsearch/blob/main/internal/rag/rag.go"
click node_faithfulness "https://github.com/soummo19/prismsearch/blob/main/internal/rag/faithfulness.go"
click node_semantic_cache "https://github.com/soummo19/prismsearch/blob/main/internal/semcache/semcache.go"
click node_response_cache "https://github.com/soummo19/prismsearch/blob/main/internal/cache/cache.go"
click node_ml_client "https://github.com/soummo19/prismsearch/blob/main/internal/mlclient/mlclient.go"
click node_ml_api "https://github.com/soummo19/prismsearch/blob/main/mlservice/app.py"
click node_embedder "https://github.com/soummo19/prismsearch/blob/main/mlservice/embedder.py"
click node_reranker "https://github.com/soummo19/prismsearch/blob/main/mlservice/reranker.py"
click node_opensearch "https://github.com/soummo19/prismsearch/blob/main/internal/opensearch/opensearch.go"
click node_qdrant "https://github.com/soummo19/prismsearch/blob/main/internal/qdrant/qdrant.go"
click node_kafka_client "https://github.com/soummo19/prismsearch/blob/main/internal/kafka/kafka.go"
click node_metrics "https://github.com/soummo19/prismsearch/blob/main/internal/obs/metrics.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_redpanda,node_normalizer_cmd,node_normalizer,node_canonical,node_priority_topics,node_indexer,node_idempotency,node_query_user toneBlue
class node_query,node_fusion,node_rag,node_faithfulness,node_semantic_cache,node_response_cache,node_answer,node_postgres toneAmber
class node_ml_client,node_ml_api,node_embedder,node_reranker toneMint
class node_opensearch,node_qdrant,node_kafka_client,node_metrics toneRose
class node_debezium toneIndigo
```

## Components

### Change ingestion

- **Redpanda**
- [Normalizer service](https://github.com/soummo19/prismsearch/blob/main/cmd/normalizer/main.go): `cmd/normalizer/main.go`
- [CDC normalizer](https://github.com/soummo19/prismsearch/blob/main/internal/normalizer/normalizer.go): `internal/normalizer/normalizer.go`
- [Canonical document](https://github.com/soummo19/prismsearch/blob/main/internal/contract/doc.go): `internal/contract/doc.go`
- **Fast / bulk topics**
- [Indexer service](https://github.com/soummo19/prismsearch/blob/main/cmd/indexer/main.go): `cmd/indexer/main.go`
- [Version guard](https://github.com/soummo19/prismsearch/blob/main/internal/idempotency/guard.go): `internal/idempotency/guard.go`

### Retrieval and RAG

- [Query API](https://github.com/soummo19/prismsearch/blob/main/cmd/query/main.go): `cmd/query/main.go`
- [Hybrid fusion](https://github.com/soummo19/prismsearch/blob/main/internal/retrieve/fusion.go): `internal/retrieve/fusion.go`
- [Cited RAG](https://github.com/soummo19/prismsearch/blob/main/internal/rag/rag.go): `internal/rag/rag.go`
- [Faithfulness guard](https://github.com/soummo19/prismsearch/blob/main/internal/rag/faithfulness.go): `internal/rag/faithfulness.go`
- [Semantic cache](https://github.com/soummo19/prismsearch/blob/main/internal/semcache/semcache.go): `internal/semcache/semcache.go`
- [Response cache](https://github.com/soummo19/prismsearch/blob/main/internal/cache/cache.go): `internal/cache/cache.go`
- **Cited answer**

### ML inference

- [ML client](https://github.com/soummo19/prismsearch/blob/main/internal/mlclient/mlclient.go): `internal/mlclient/mlclient.go`
- [ML HTTP service](https://github.com/soummo19/prismsearch/blob/main/mlservice/app.py): `mlservice/app.py`
- [Embedding backend](https://github.com/soummo19/prismsearch/blob/main/mlservice/embedder.py): `mlservice/embedder.py`
- [Reranker backend](https://github.com/soummo19/prismsearch/blob/main/mlservice/reranker.py): `mlservice/reranker.py`

### Stores and telemetry

- [OpenSearch BM25](https://github.com/soummo19/prismsearch/blob/main/internal/opensearch/opensearch.go): `internal/opensearch/opensearch.go`
- [Qdrant vectors](https://github.com/soummo19/prismsearch/blob/main/internal/qdrant/qdrant.go): `internal/qdrant/qdrant.go`
- [Kafka client](https://github.com/soummo19/prismsearch/blob/main/internal/kafka/kafka.go): `internal/kafka/kafka.go`
- [Metrics](https://github.com/soummo19/prismsearch/blob/main/internal/obs/metrics.go): `internal/obs/metrics.go`

### Other components

- **PostgreSQL source**
- **Debezium CDC**
- **User**

## Connections

- PostgreSQL source → Debezium CDC: streams changes (evidence: `README.md`)
- Debezium CDC → Redpanda: publishes CDC (evidence: `README.md`)
- Redpanda → Normalizer service: feeds events (evidence: `cmd/normalizer/main.go`)
- Normalizer service → CDC normalizer: parses events (evidence: `cmd/normalizer/main.go`)
- CDC normalizer → Canonical document: creates documents (evidence: `internal/normalizer/normalizer.go`)
- Normalizer service → Fast / bulk topics: routes by tier (evidence: `cmd/normalizer/main.go`)
- Fast / bulk topics → Indexer service: feeds lanes (evidence: `README.md`)
- Indexer service → Version guard: checks versions (evidence: `README.md`)
- Indexer service → ML client: requests embeddings (evidence: `cmd/indexer/main.go`)
- Indexer service → OpenSearch BM25: writes documents (evidence: `cmd/indexer/main.go`)
- Indexer service → Qdrant vectors: writes vectors (evidence: `cmd/indexer/main.go`)
- User → Query API: submits search (evidence: `README.md`)
- Query API → OpenSearch BM25: runs BM25 (evidence: `README.md`)
- Query API → Qdrant vectors: runs vector search (evidence: `README.md`)
- Query API → Hybrid fusion: combines rankings (evidence: `README.md`)
- Query API → ML client: requests reranking (evidence: `cmd/query/main.go`)
- Query API → Semantic cache: checks semantic match (evidence: `cmd/query/main.go`)
- Query API → Response cache: checks cached response (evidence: `cmd/query/main.go`)
- Query API → Cited RAG: generates answer (evidence: `README.md`)
- Cited RAG → Faithfulness guard: checks grounding (evidence: `README.md`)
- Cited RAG → Cited answer: returns citations (evidence: `README.md`)
- Cited answer → User: returns answer (evidence: `README.md`)
- ML client → ML HTTP service: calls HTTP endpoints (evidence: `internal/mlclient/mlclient.go`)
- ML HTTP service → Embedding backend: embeds texts (evidence: `mlservice/app.py`)
- ML HTTP service → Reranker backend: scores documents (evidence: `mlservice/app.py`)
- Indexer service → Metrics: records freshness (evidence: `README.md`)
