How pg-herd/pgex works
pgex is a PostgreSQL table and filtered-row transfer CLI. A run resolves CLI/YAML settings, discovers or selects source tables, plans filters, reads through snapshot-scoped source sessions, and schedules destination transfers; target options also govern layout checks, locking, deletion, and truncation. The graph preserves these distinct stages and the connection/worker layer. Some subsystem links rely on README or index evidence because their implementations were not excerpted; those uncertain links are dashed.
Main entry points: worker.rs Transfer workers · runner.rs Job runner · args.rs CLI arguments · postgres.rs Postgres target
Components 20
Inputs and planning
- CLI arguments
src/cli/args.rs - YAML loader
src/config/loader/yaml.rs - Config compiler
src/config/compiler/compile.rs - Predicate parsing
src/predicate/parser.rs - Job runner
src/job_runner/runner.rs
Source consistency
- Source catalog
src/catalog/reader.rs - Table layouts
src/catalog/layout.rs - Table planner
src/plan/planner.rs - Snapshot lease
src/snapshot/lease.rs - Snapshot readers
src/snapshot/session.rs
Transfer execution
- Postgres target
src/targets/postgres.rs - Transfer workers
src/targets/postgres/worker.rs - Row transfer
src/targets/postgres/transfer.rs - Truncate mode
src/targets/postgres/truncate.rs - Commit groups
src/targets/postgres/commit_group.rs - Work queue
src/work_queue/queue.rs
Database runtime
- Postgres sessions
src/connection/client.rs
Other components
- CLI user
- Source PostgreSQL
- Destination PostgreSQL
Connections 25
- CLI user → CLI arguments: provides options
src/cli/args.rs - CLI arguments → Job runner: starts run
- CLI arguments → Predicate parsing: parses filters
src/cli/args.rs - CLI arguments → YAML loader: selects config
- YAML loader → Config compiler: loads settings
- Job runner → Config compiler: compiles settings
- Job runner → Source catalog: creates catalog reader
- Source catalog → Source PostgreSQL: reads metadata
- Source catalog → Table layouts: reads layouts
src/catalog/layout.rs - Job runner → Snapshot lease: creates lease
- Job runner → Postgres target: configures target
- Job runner → Table planner: plans source tables
- Table planner → Source catalog: uses table catalog
src/plan/planner.rs - Postgres target → Table planner: accepts source plan
src/targets/mod.rs - Postgres target → Snapshot lease: uses snapshot readers
- Postgres target → Transfer workers: schedules workers
src/targets/postgres.rs - Transfer workers → Snapshot readers: opens source reader
src/targets/postgres/worker.rs - Transfer workers → Postgres sessions: opens destination session
src/targets/postgres/worker.rs - Transfer workers → Row transfer: transfers rows
- Transfer workers → Truncate mode: runs truncate mode
src/targets/postgres/worker.rs - Transfer workers → Commit groups: transfers groups
src/targets/postgres/worker.rs - Transfer workers → Work queue: consumes work
src/targets/postgres/worker.rs - Snapshot readers → Source PostgreSQL: reads snapshot data
- Postgres sessions → Destination PostgreSQL: executes queries
- Postgres sessions → Source PostgreSQL: connects to source