# pg-herd/pgex architecture

> Architecture diagram of the GitHub repository pg-herd/pgex, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pg-herd/pgex
- Repository: https://github.com/pg-herd/pgex

## Overview

pgex is a PostgreSQL table and filtered-row transfer CLI. A run resolves CLI/YAML settings, discovers or selects source tables, plans filters, reads through snapshot-scoped source sessions, and schedules destination transfers; target options also govern layout checks, locking, deletion, and truncation. The graph preserves these distinct stages and the connection/worker layer. Some subsystem links rely on README or index evidence because their implementations were not excerpted; those uncertain links are dashed.

Main entry points: [`src/targets/postgres/worker.rs`](https://github.com/pg-herd/pgex/blob/main/src/targets/postgres/worker.rs) (Transfer workers), [`src/job_runner/runner.rs`](https://github.com/pg-herd/pgex/blob/main/src/job_runner/runner.rs) (Job runner), [`src/cli/args.rs`](https://github.com/pg-herd/pgex/blob/main/src/cli/args.rs) (CLI arguments), [`src/targets/postgres.rs`](https://github.com/pg-herd/pgex/blob/main/src/targets/postgres.rs) (Postgres target)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inputs["Inputs and planning"]
  node_cli["CLI arguments<br/>[args.rs]"]
  node_yaml["YAML loader<br/>[yaml.rs]"]
  node_config["Config compiler<br/>[compile.rs]"]
  node_predicates["Predicate parsing<br/>[parser.rs]"]
  node_runner["Job runner<br/>[runner.rs]"]
end

subgraph group_source["Source consistency"]
  node_catalog["Source catalog<br/>[reader.rs]"]
  node_layout["Table layouts<br/>[layout.rs]"]
  node_planner["Table planner<br/>[planner.rs]"]
  node_snapshot["Snapshot lease<br/>[lease.rs]"]
  node_readers["Snapshot readers<br/>[session.rs]"]
end

subgraph group_transfer["Transfer execution"]
  node_target["Postgres target<br/>[postgres.rs]"]
  node_workers["Transfer workers<br/>[worker.rs]"]
  node_copy["Row transfer<br/>[transfer.rs]"]
  node_truncate["Truncate mode<br/>[truncate.rs]"]
  node_commit["Commit groups<br/>[commit_group.rs]"]
  node_queue["Work queue<br/>[queue.rs]"]
end

subgraph group_infra["Database runtime"]
  node_connection["Postgres sessions<br/>[client.rs]"]
end

node_user(("CLI user"))
node_sourcepg[("Source PostgreSQL")]
node_targetpg[("Destination PostgreSQL")]

node_user -->|"provides options"| node_cli
node_cli -->|"starts run"| node_runner
node_cli -->|"parses filters"| node_predicates
node_cli -->|"selects config"| node_yaml
node_yaml -.->|"loads settings"| node_config
node_runner -->|"compiles settings"| node_config
node_runner -->|"creates catalog reader"| node_catalog
node_catalog -.->|"reads metadata"| node_sourcepg
node_catalog -->|"reads layouts"| node_layout
node_runner -->|"creates lease"| node_snapshot
node_runner -->|"configures target"| node_target
node_runner -.->|"plans source tables"| node_planner
node_planner -->|"uses table catalog"| node_catalog
node_target -->|"accepts source plan"| node_planner
node_target -->|"uses snapshot readers"| node_snapshot
node_target -->|"schedules workers"| node_workers
node_workers -->|"opens source reader"| node_readers
node_workers -->|"opens destination session"| node_connection
node_workers -.->|"transfers rows"| node_copy
node_workers -->|"runs truncate mode"| node_truncate
node_workers -->|"transfers groups"| node_commit
node_workers -->|"consumes work"| node_queue
node_readers -.->|"reads snapshot data"| node_sourcepg
node_connection -.->|"executes queries"| node_targetpg
node_connection -.->|"connects to source"| node_sourcepg

click node_cli "https://github.com/pg-herd/pgex/blob/main/src/cli/args.rs"
click node_yaml "https://github.com/pg-herd/pgex/blob/main/src/config/loader/yaml.rs"
click node_config "https://github.com/pg-herd/pgex/blob/main/src/config/compiler/compile.rs"
click node_predicates "https://github.com/pg-herd/pgex/blob/main/src/predicate/parser.rs"
click node_runner "https://github.com/pg-herd/pgex/blob/main/src/job_runner/runner.rs"
click node_catalog "https://github.com/pg-herd/pgex/blob/main/src/catalog/reader.rs"
click node_layout "https://github.com/pg-herd/pgex/blob/main/src/catalog/layout.rs"
click node_planner "https://github.com/pg-herd/pgex/blob/main/src/plan/planner.rs"
click node_snapshot "https://github.com/pg-herd/pgex/blob/main/src/snapshot/lease.rs"
click node_readers "https://github.com/pg-herd/pgex/blob/main/src/snapshot/session.rs"
click node_target "https://github.com/pg-herd/pgex/blob/main/src/targets/postgres.rs"
click node_workers "https://github.com/pg-herd/pgex/blob/main/src/targets/postgres/worker.rs"
click node_copy "https://github.com/pg-herd/pgex/blob/main/src/targets/postgres/transfer.rs"
click node_truncate "https://github.com/pg-herd/pgex/blob/main/src/targets/postgres/truncate.rs"
click node_commit "https://github.com/pg-herd/pgex/blob/main/src/targets/postgres/commit_group.rs"
click node_queue "https://github.com/pg-herd/pgex/blob/main/src/work_queue/queue.rs"
click node_connection "https://github.com/pg-herd/pgex/blob/main/src/connection/client.rs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_yaml,node_config,node_predicates,node_runner,node_user toneBlue
class node_catalog,node_layout,node_planner,node_snapshot,node_readers,node_sourcepg,node_targetpg toneAmber
class node_target,node_workers,node_copy,node_truncate,node_commit,node_queue toneMint
class node_connection toneRose
```

## Components

### Inputs and planning

- [CLI arguments](https://github.com/pg-herd/pgex/blob/main/src/cli/args.rs): `src/cli/args.rs`
- [YAML loader](https://github.com/pg-herd/pgex/blob/main/src/config/loader/yaml.rs): `src/config/loader/yaml.rs`
- [Config compiler](https://github.com/pg-herd/pgex/blob/main/src/config/compiler/compile.rs): `src/config/compiler/compile.rs`
- [Predicate parsing](https://github.com/pg-herd/pgex/blob/main/src/predicate/parser.rs): `src/predicate/parser.rs`
- [Job runner](https://github.com/pg-herd/pgex/blob/main/src/job_runner/runner.rs): `src/job_runner/runner.rs`

### Source consistency

- [Source catalog](https://github.com/pg-herd/pgex/blob/main/src/catalog/reader.rs): `src/catalog/reader.rs`
- [Table layouts](https://github.com/pg-herd/pgex/blob/main/src/catalog/layout.rs): `src/catalog/layout.rs`
- [Table planner](https://github.com/pg-herd/pgex/blob/main/src/plan/planner.rs): `src/plan/planner.rs`
- [Snapshot lease](https://github.com/pg-herd/pgex/blob/main/src/snapshot/lease.rs): `src/snapshot/lease.rs`
- [Snapshot readers](https://github.com/pg-herd/pgex/blob/main/src/snapshot/session.rs): `src/snapshot/session.rs`

### Transfer execution

- [Postgres target](https://github.com/pg-herd/pgex/blob/main/src/targets/postgres.rs): `src/targets/postgres.rs`
- [Transfer workers](https://github.com/pg-herd/pgex/blob/main/src/targets/postgres/worker.rs): `src/targets/postgres/worker.rs`
- [Row transfer](https://github.com/pg-herd/pgex/blob/main/src/targets/postgres/transfer.rs): `src/targets/postgres/transfer.rs`
- [Truncate mode](https://github.com/pg-herd/pgex/blob/main/src/targets/postgres/truncate.rs): `src/targets/postgres/truncate.rs`
- [Commit groups](https://github.com/pg-herd/pgex/blob/main/src/targets/postgres/commit_group.rs): `src/targets/postgres/commit_group.rs`
- [Work queue](https://github.com/pg-herd/pgex/blob/main/src/work_queue/queue.rs): `src/work_queue/queue.rs`

### Database runtime

- [Postgres sessions](https://github.com/pg-herd/pgex/blob/main/src/connection/client.rs): `src/connection/client.rs`

### Other components

- **CLI user**
- **Source PostgreSQL**
- **Destination PostgreSQL**

## Connections

- CLI user → CLI arguments: provides options (evidence: `src/cli/args.rs`)
- CLI arguments → Job runner: starts run
- CLI arguments → Predicate parsing: parses filters (evidence: `src/cli/args.rs`)
- CLI arguments → YAML loader: selects config
- YAML loader → Config compiler: loads settings
- Job runner → Config compiler: compiles settings
- Job runner → Source catalog: creates catalog reader
- Source catalog → Source PostgreSQL: reads metadata
- Source catalog → Table layouts: reads layouts (evidence: `src/catalog/layout.rs`)
- Job runner → Snapshot lease: creates lease
- Job runner → Postgres target: configures target
- Job runner → Table planner: plans source tables
- Table planner → Source catalog: uses table catalog (evidence: `src/plan/planner.rs`)
- Postgres target → Table planner: accepts source plan (evidence: `src/targets/mod.rs`)
- Postgres target → Snapshot lease: uses snapshot readers
- Postgres target → Transfer workers: schedules workers (evidence: `src/targets/postgres.rs`)
- Transfer workers → Snapshot readers: opens source reader (evidence: `src/targets/postgres/worker.rs`)
- Transfer workers → Postgres sessions: opens destination session (evidence: `src/targets/postgres/worker.rs`)
- Transfer workers → Row transfer: transfers rows
- Transfer workers → Truncate mode: runs truncate mode (evidence: `src/targets/postgres/worker.rs`)
- Transfer workers → Commit groups: transfers groups (evidence: `src/targets/postgres/worker.rs`)
- Transfer workers → Work queue: consumes work (evidence: `src/targets/postgres/worker.rs`)
- Snapshot readers → Source PostgreSQL: reads snapshot data
- Postgres sessions → Destination PostgreSQL: executes queries
- Postgres sessions → Source PostgreSQL: connects to source
