# janbjorge/pgqueuer architecture

> Architecture diagram of the GitHub repository janbjorge/pgqueuer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/janbjorge/pgqueuer
- Repository: https://github.com/janbjorge/pgqueuer

## Overview

##### Project type and purpose
- PGQueuer is an open-source Python library and CLI tool that provides a high-performance job queue built on PostgreSQL. It exposes both a programmatic API (the `pgqueuer` package) and a command-line interface (`pgq`) to enqueue/dequeue jobs, schedule recurring tasks, and display a real-time dashboard.

##### File structure
- Top-level CLI & orchestration:
- Dockerfile, docker-compose.yml (containerized Postgres + optional services)
- Makefile (build/test tasks)
- Core library (`pgqueuer/`):
- `db.py`, `drivers.py` – abstract driver & Postgres driver implementations
- `queries.py`, `qb.py` – SQL query generation & execution
- `executors.py`, `tm.py`, `sm.py`, `qm.py` – thread/process executors, task manager, state machine
- `heartbeat.py`, `listeners.py`, `completion.py` – event/listener loops, status watchers
- `buffers.py`, `cache.py` – in-memory job buffering & caching
- `cli.py`, `applications.py`, `factories.py` – command-line commands & factory functions
- `tracing.py`, `logconfig.py` – optional distributed tracing & logging setup
- Support files: examples/, docs/, tools/, test/

##### Insights from README
- Uses PostgreSQL’s LISTEN/NOTIFY for real-time wake-ups
- Uses `FOR UPDATE SKIP LOCKED` for safe concurrency
- Supports recurring schedules (cron syntax) and graceful shutdown
- Offers a live dashboard (CLI) and integration points for metrics (Prometheus)

## Diagram (Mermaid)

```mermaid
flowchart TB
  %% Application Layer
  subgraph "Application Layer"
    direction TB
    ClientApp["Client Application(s)"]:::yellow
    CLI["CLI Interface (pgq)"]:::yellow
  end

  %% Core Library
  subgraph "Core Library"
    direction TB
    Driver["Driver Layer\n(db.py, factories.py)"]:::core
    QueryMgr["Query Manager & Builder\n(queries.py, qb.py)"]:::core
    Listener["Listener/Notifier Loop\n(listeners.py)"]:::core
    Executor["Executor Pool & Task Manager\n(executors.py, tm.py)"]:::core
    StateMachine["State Machine & Workflow\n(sm.py, qm.py)"]:::core
    Scheduler["Scheduler (recurring tasks)\n(examples/scheduler.py)"]:::core
    Heartbeat["Heartbeat & Completion Watcher\n(heartbeat.py, completion.py)"]:::core
    BufferCache["Buffers & Cache\n(buffers.py, cache.py)"]:::core
    LogTrace["Logging & Tracing\n(logconfig.py, tracing.py)"]:::core
    CoreEntry["Core Entry Points\n(__init__.py, __main__.py)"]:::core
  end

  %% Infrastructure & External
  subgraph "Infrastructure & Integrations"
    direction TB
    Postgres[(PostgreSQL Database)]:::external
    PromExporter["Prometheus Metrics Exporter"]:::external
    DockerInfra["Docker & Orchestration\n(Dockerfile, docker-compose.yml)"]:::infra
    Examples["Examples & Integrations\n(examples/)"]:::infra
    Automation["Build/Test Automation\n(Makefile, test/)"]:::infra
  end

  %% Data Flows
  ClientApp -->|"import pgqueuer API"| QueryMgr
  CLI -->|"CLI commands"| CoreEntry
  CoreEntry --> Driver
  CoreEntry --> QueryMgr

  %% Job enqueue flow
  QueryMgr -->|"1. INSERT + NOTIFY"| Postgres
  Postgres -->|"2. NOTIFY"| Listener
  Listener -->|"wake up"| Executor
  Executor -->|"3. SELECT FOR UPDATE SKIP LOCKED"| Postgres
  Executor -->|"execute job entrypoint"| ClientApp
  Executor -->|"job completion"| Heartbeat
  Heartbeat -->|"UPDATE status"| Postgres
  Heartbeat -->|"record metrics"| PromExporter

  %% Scheduler flow
  Scheduler -->|"5. enqueue recurring jobs"| QueryMgr

  %% Dependencies among core modules
  Driver --> QueryMgr
  BufferCache --> Executor
  LogTrace --> Executor
  StateMachine --> Executor
  StateMachine --> Heartbeat

  %% Orchestration & examples
  DockerInfra --> Postgres
  PromExporter --> DockerInfra
  Examples --> ClientApp
  Automation --> CoreEntry

  %% Click events
  click CLI "https://github.com/janbjorge/pgqueuer/blob/main/pgqueuer/cli.py"
  click CoreEntry "https://github.com/janbjorge/pgqueuer/blob/main/pgqueuer/__main__.py"
  click Driver "https://github.com/janbjorge/pgqueuer/blob/main/pgqueuer/db.py"
  click Driver "https://github.com/janbjorge/pgqueuer/blob/main/pgqueuer/factories.py"
  click QueryMgr "https://github.com/janbjorge/pgqueuer/blob/main/pgqueuer/queries.py"
  click QueryMgr "https://github.com/janbjorge/pgqueuer/blob/main/pgqueuer/qb.py"
  click QueryMgr "https://github.com/janbjorge/pgqueuer/blob/main/pgqueuer/query_helpers.py"
  click Listener "https://github.com/janbjorge/pgqueuer/blob/main/pgqueuer/listeners.py"
  click Executor "https://github.com/janbjorge/pgqueuer/blob/main/pgqueuer/executors.py"
  click Executor "https://github.com/janbjorge/pgqueuer/blob/main/pgqueuer/tm.py"
  click StateMachine "https://github.com/janbjorge/pgqueuer/blob/main/pgqueuer/sm.py"
  click StateMachine "https://github.com/janbjorge/pgqueuer/blob/main/pgqueuer/qm.py"
  click Scheduler "https://github.com/janbjorge/pgqueuer/blob/main/examples/scheduler.py"
  click Heartbeat "https://github.com/janbjorge/pgqueuer/blob/main/pgqueuer/heartbeat.py"
  click Heartbeat "https://github.com/janbjorge/pgqueuer/blob/main/pgqueuer/completion.py"
  click BufferCache "https://github.com/janbjorge/pgqueuer/blob/main/pgqueuer/buffers.py"
  click BufferCache "https://github.com/janbjorge/pgqueuer/blob/main/pgqueuer/cache.py"
  click LogTrace "https://github.com/janbjorge/pgqueuer/blob/main/pgqueuer/logconfig.py"
  click LogTrace "https://github.com/janbjorge/pgqueuer/blob/main/pgqueuer/tracing.py"
  click DockerInfra "https://github.com/janbjorge/pgqueuer/tree/main/Dockerfile"
  click DockerInfra "https://github.com/janbjorge/pgqueuer/blob/main/docker-compose.yml"
  click PromExporter "https://github.com/janbjorge/pgqueuer/blob/main/tools/prometheus/prometheus.py"
  click PromExporter "https://github.com/janbjorge/pgqueuer/tree/main/tools/prometheus/Dockerfile"
  click Examples "https://github.com/janbjorge/pgqueuer/blob/main/examples/consumer.py"
  click Examples "https://github.com/janbjorge/pgqueuer/blob/main/examples/producer.py"
  click Examples "https://github.com/janbjorge/pgqueuer/blob/main/examples/fastapi_usage.py"
  click Examples "https://github.com/janbjorge/pgqueuer/blob/main/examples/flask_sync_usage.py"
  click Automation "https://github.com/janbjorge/pgqueuer/tree/main/Makefile"
  click Automation "https://github.com/janbjorge/pgqueuer/tree/main/test/"

  %% Styles
  classDef external fill:#a8f0a8,stroke:#0a0,stroke-width:1px
  classDef core fill:#a0c4ff,stroke:#0366d6,stroke-width:1px
  classDef yellow fill:#fff7a8,stroke:#d6a100,stroke-width:1px
  classDef infra fill:#e0e0e0,stroke:#888,stroke-width:1px
  class ClientApp,CLI yellow
  class Driver,QueryMgr,Listener,Executor,StateMachine,Scheduler,Heartbeat,BufferCache,LogTrace,CoreEntry core
  class Postgres,PromExporter external
  class DockerInfra,Examples,Automation infra
```
