# toluaina/pgsync architecture

> Architecture diagram of the GitHub repository toluaina/pgsync, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/toluaina/pgsync
- Repository: https://github.com/toluaina/pgsync

## Overview

PGSync keeps PostgreSQL, MySQL, or MariaDB authoritative while producing denormalized Elasticsearch/OpenSearch documents. An operator starts the sync with a JSON schema; setup configures CDC, then Sync consumes changes or polls, rebuilds affected relational documents through QueryBuilder, applies transforms and plugins, and bulk-indexes results. Checkpoints and Redis support recovery and queueing. The graph preserves optional polling, masking/slug enrichment, and embedding integrations; unsampled modules are represented only where their paths are explicit.

Main entry points: [`pgsync/sync.py`](https://github.com/toluaina/pgsync/blob/main/pgsync/sync.py) (Sync Orchestrator), [`pgsync/plugin.py`](https://github.com/toluaina/pgsync/blob/main/pgsync/plugin.py) (Plugin Runtime), [`pgsync/base.py`](https://github.com/toluaina/pgsync/blob/main/pgsync/base.py) (Database Adapter), [`pgsync/trigger.py`](https://github.com/toluaina/pgsync/blob/main/pgsync/trigger.py) (CDC Triggers)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_source_cdc["Source &amp; CDC"]
  node_source_db[("Source Database")]
  node_sync["Sync Orchestrator<br/>[sync.py]"]
  node_db_adapter["Database Adapter<br/>[base.py]"]
  node_cdc_triggers["CDC Triggers<br/>[trigger.py]"]
  node_change_queue["Redis Queue<br/>[redisqueue.py]"]
end

subgraph group_document_pipeline["Document Pipeline"]
  node_schema_tree["Schema Tree<br/>[node.py]"]
  node_query_builder["Query Builder<br/>[querybuilder.py]"]
  node_transformer["Document Transforms<br/>[transform.py]"]
end

subgraph group_delivery_state["Delivery &amp; State"]
  node_sink["Index Sink<br/>[sink.py]"]
  node_search_client["Search Client<br/>[search_client.py]"]
  node_search_engine[("Search Engine")]
  node_checkpoint[("Checkpoint Store<br/>[sync.py]")]
end

subgraph group_extensions["Extensions"]
  node_plugin_runtime["Plugin Runtime<br/>[plugin.py]"]
  node_masking_plugin["Masking Plugin<br/>[masking_plugin.py]"]
  node_slug_plugin["Slug Plugin<br/>[slug_plugin.py]"]
  node_embedding_plugin["Embedding Plugin"]
  node_embedding_service["Embedding Service"]
end

node_operator(("Operator"))

node_operator -->|"starts"| node_sync
node_sync -->|"loads schema"| node_schema_tree
node_sync -->|"configures CDC"| node_cdc_triggers
node_cdc_triggers -->|"creates triggers"| node_source_db
node_sync -->|"reads changes"| node_db_adapter
node_db_adapter -->|"captures changes"| node_source_db
node_sync -.->|"queues events"| node_change_queue
node_sync -->|"builds queries"| node_query_builder
node_db_adapter -->|"executes queries"| node_source_db
node_sync -->|"transforms documents"| node_transformer
node_transformer -->|"dispatches plugins"| node_plugin_runtime
node_plugin_runtime -->|"runs masking"| node_masking_plugin
node_plugin_runtime -->|"runs slugging"| node_slug_plugin
node_plugin_runtime -.->|"runs embedding"| node_embedding_plugin
node_embedding_plugin -.->|"requests embeddings"| node_embedding_service
node_sync -->|"writes documents"| node_sink
node_sink -->|"uses client"| node_search_client
node_search_client -->|"bulk indexes"| node_search_engine
node_sync -->|"saves checkpoint"| node_checkpoint
node_sync -.->|"stores checkpoint"| node_change_queue

click node_sync "https://github.com/toluaina/pgsync/blob/main/pgsync/sync.py"
click node_db_adapter "https://github.com/toluaina/pgsync/blob/main/pgsync/base.py"
click node_cdc_triggers "https://github.com/toluaina/pgsync/blob/main/pgsync/trigger.py"
click node_change_queue "https://github.com/toluaina/pgsync/blob/main/pgsync/redisqueue.py"
click node_schema_tree "https://github.com/toluaina/pgsync/blob/main/pgsync/node.py"
click node_query_builder "https://github.com/toluaina/pgsync/blob/main/pgsync/querybuilder.py"
click node_transformer "https://github.com/toluaina/pgsync/blob/main/pgsync/transform.py"
click node_sink "https://github.com/toluaina/pgsync/blob/main/pgsync/sink.py"
click node_search_client "https://github.com/toluaina/pgsync/blob/main/pgsync/search_client.py"
click node_checkpoint "https://github.com/toluaina/pgsync/blob/main/pgsync/sync.py"
click node_plugin_runtime "https://github.com/toluaina/pgsync/blob/main/pgsync/plugin.py"
click node_masking_plugin "https://github.com/toluaina/pgsync/blob/main/plugins/masking_plugin.py"
click node_slug_plugin "https://github.com/toluaina/pgsync/blob/main/plugins/slug_plugin.py"
click node_embedding_plugin "https://github.com/toluaina/pgsync/blob/main/plugins/anthropic_plugin.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_source_db,node_sync,node_db_adapter,node_cdc_triggers,node_change_queue toneBlue
class node_schema_tree,node_query_builder,node_transformer toneAmber
class node_sink,node_search_client,node_search_engine,node_checkpoint toneMint
class node_plugin_runtime,node_masking_plugin,node_slug_plugin,node_embedding_plugin,node_embedding_service toneRose
class node_operator toneIndigo
```

## Components

### Source & CDC

- **Source Database**
- [Sync Orchestrator](https://github.com/toluaina/pgsync/blob/main/pgsync/sync.py): `pgsync/sync.py`
- [Database Adapter](https://github.com/toluaina/pgsync/blob/main/pgsync/base.py): `pgsync/base.py`
- [CDC Triggers](https://github.com/toluaina/pgsync/blob/main/pgsync/trigger.py): `pgsync/trigger.py`
- [Redis Queue](https://github.com/toluaina/pgsync/blob/main/pgsync/redisqueue.py): `pgsync/redisqueue.py`

### Document Pipeline

- [Schema Tree](https://github.com/toluaina/pgsync/blob/main/pgsync/node.py): `pgsync/node.py`
- [Query Builder](https://github.com/toluaina/pgsync/blob/main/pgsync/querybuilder.py): `pgsync/querybuilder.py`
- [Document Transforms](https://github.com/toluaina/pgsync/blob/main/pgsync/transform.py): `pgsync/transform.py`

### Delivery & State

- [Index Sink](https://github.com/toluaina/pgsync/blob/main/pgsync/sink.py): `pgsync/sink.py`
- [Search Client](https://github.com/toluaina/pgsync/blob/main/pgsync/search_client.py): `pgsync/search_client.py`
- **Search Engine**
- [Checkpoint Store](https://github.com/toluaina/pgsync/blob/main/pgsync/sync.py): `pgsync/sync.py`

### Extensions

- [Plugin Runtime](https://github.com/toluaina/pgsync/blob/main/pgsync/plugin.py): `pgsync/plugin.py`
- [Masking Plugin](https://github.com/toluaina/pgsync/blob/main/plugins/masking_plugin.py): `plugins/masking_plugin.py`
- [Slug Plugin](https://github.com/toluaina/pgsync/blob/main/plugins/slug_plugin.py): `plugins/slug_plugin.py`
- [Embedding Plugin](https://github.com/toluaina/pgsync/blob/main/plugins/anthropic_plugin.py): `plugins/anthropic_plugin.py`
- **Embedding Service**

### Other components

- **Operator**

## Connections

- Operator → Sync Orchestrator: starts
- Sync Orchestrator → Schema Tree: loads schema
- Sync Orchestrator → CDC Triggers: configures CDC
- CDC Triggers → Source Database: creates triggers
- Sync Orchestrator → Database Adapter: reads changes
- Database Adapter → Source Database: captures changes
- Sync Orchestrator → Redis Queue: queues events
- Sync Orchestrator → Query Builder: builds queries
- Database Adapter → Source Database: executes queries
- Sync Orchestrator → Document Transforms: transforms documents
- Document Transforms → Plugin Runtime: dispatches plugins
- Plugin Runtime → Masking Plugin: runs masking
- Plugin Runtime → Slug Plugin: runs slugging
- Plugin Runtime → Embedding Plugin: runs embedding
- Embedding Plugin → Embedding Service: requests embeddings
- Sync Orchestrator → Index Sink: writes documents
- Index Sink → Search Client: uses client
- Search Client → Search Engine: bulk indexes
- Sync Orchestrator → Checkpoint Store: saves checkpoint
- Sync Orchestrator → Redis Queue: stores checkpoint
