How toluaina/pgsync works
PGSync keeps PostgreSQL, MySQL, or MariaDB authoritative while producing denormalized Elasticsearch/OpenSearch documents. An operator starts the sync with a JSON schema; setup configures CDC, then Sync consumes changes or polls, rebuilds affected relational documents through QueryBuilder, applies transforms and plugins, and bulk-indexes results. Checkpoints and Redis support recovery and queueing. The graph preserves optional polling, masking/slug enrichment, and embedding integrations; unsampled modules are represented only where their paths are explicit.
Main entry points: sync.py Sync Orchestrator · plugin.py Plugin Runtime · base.py Database Adapter · trigger.py CDC Triggers
Components 18
Source & CDC
- Source Database
- Sync Orchestrator
pgsync/sync.py - Database Adapter
pgsync/base.py - CDC Triggers
pgsync/trigger.py - Redis Queue
pgsync/redisqueue.py
Document Pipeline
- Schema Tree
pgsync/node.py - Query Builder
pgsync/querybuilder.py - Document Transforms
pgsync/transform.py
Delivery & State
- Index Sink
pgsync/sink.py - Search Client
pgsync/search_client.py - Search Engine
- Checkpoint Store
pgsync/sync.py
Extensions
- Plugin Runtime
pgsync/plugin.py - Masking Plugin
plugins/masking_plugin.py - Slug Plugin
plugins/slug_plugin.py - Embedding Plugin
plugins/anthropic_plugin.py - Embedding Service
Other components
- Operator
Connections 20
- Operator → Sync Orchestrator: starts
- Sync Orchestrator → Schema Tree: loads schema
- Sync Orchestrator → CDC Triggers: configures CDC
- CDC Triggers → Source Database: creates triggers
- Sync Orchestrator → Database Adapter: reads changes
- Database Adapter → Source Database: captures changes
- Sync Orchestrator → Redis Queue: queues events
- Sync Orchestrator → Query Builder: builds queries
- Database Adapter → Source Database: executes queries
- Sync Orchestrator → Document Transforms: transforms documents
- Document Transforms → Plugin Runtime: dispatches plugins
- Plugin Runtime → Masking Plugin: runs masking
- Plugin Runtime → Slug Plugin: runs slugging
- Plugin Runtime → Embedding Plugin: runs embedding
- Embedding Plugin → Embedding Service: requests embeddings
- Sync Orchestrator → Index Sink: writes documents
- Index Sink → Search Client: uses client
- Search Client → Search Engine: bulk indexes
- Sync Orchestrator → Checkpoint Store: saves checkpoint
- Sync Orchestrator → Redis Queue: stores checkpoint