# supabase/pg_net architecture

> Architecture diagram of the GitHub repository supabase/pg_net, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/supabase/pg_net
- Repository: https://github.com/supabase/pg_net

## Overview

##### Project Type and Purpose
- This is a PostgreSQL extension (“pg_net”) written in C that adds asynchronous HTTP/HTTPS capabilities to the database engine.
- It integrates deeply with PostgreSQL via a background worker process, unlogged tables for queuing, and SQL-callable functions.

##### File Structure Analysis
- src/ contains the C source files and headers (core.c, worker.c, event.c, util.c, errors.c, plus CURL and PG Prelude headers).
- pg_net.control.in and sql/ contain extension metadata and versioned SQL migrations for table creation and upgrade paths.
- test/ holds Python-based integration tests (conftest.py, test_engine.py, HTTP behavior tests, worker error tests).
- nix/ and shell.nix define the reproducible dev environment (Nix build of libcurl, PostgreSQL, Nginx config for docs).
- docs/ and mkdocs.yml drive the published documentation.
- Makefile & GitHub Actions workflow for build, test, coverage, and publish.

##### Key Components
a. SQL Client Layer
- SQL scripts, psql clients, Trigger functions (PL/pgSQL) and Cron jobs (PG_CRON) that call net.http_get/post/delete.
b. PostgreSQL Server
- Core Postgres process with `shared_preload_libraries = 'pg_net'`.
- The pg_net background worker thread/process registered via the PostgreSQL BGWorker API.
c. pg_net Extension Modules (inside PostgreSQL process)
- Producer: SQL-callable functions in PL/pgSQL or C stubs that INSERT into net.http_request_queue.
- Consumer: Background worker polling net.http_request_queue, executing requests via libcurl, and writing to net._http_response.
- Internal C components:
- core.c – extension init and function exports
- worker.c – background worker loop & scheduling
- event.c – event-loop integration
- util.c + errors.c – helper routines
d. Persistence Queues (Unlogged Tables)
- net.http_request_queue (pending requests)
- net._http_response (completed results and errors)
e. External HTTP/HTTPS Services
- Any REST API endpoints called asynchronously (webhooks, serverless functions, data sync targets).
f. Dev/CI Tooling (side components)
- Nix environment for reproducible builds
- GitHub Actions for compile/test/coverage
- MkDocs for documentation site

##### Relationships and Data Flows
- [SQL Client / Trigger / Cron] --(SQL CALL net.http_*)-> [http_request_queue]
- [pg_net Background Worker] --(poll queue)-> [http_request_queue]
- [Background Worker] --(libcurl HTTP request)-> [External HTTP Service]
- [External HTTP Service] --(HTTP response)-> [Background Worker]
- [Background Worker] --(INSERT/UPDATE response)-> [net._http_response]
- [Monitoring SQL Queries] --(SELECT)-> [net._http_response]

##### Architectural Patterns & Principles
- Producer–Consumer via database queue.
- Background worker (asynchronous processing) integrated into the database.
- Unlogged tables for high-performance queuing.
- Modular C design separating core, worker, events, utilities, and error handling.
- PL/pgSQL wrappers for user-friendly SQL API.

##### Technologies, Frameworks, Libraries
- PostgreSQL BGWorker API
- libcurl for HTTP/HTTPS
- C (ISO C) and PGXS build system (Makefile)
- PL/pgSQL for extension functions
- Python + pytest for integration tests
- Nix for reproducible dev/deployment
- GitHub Actions CI
- MkDocs for documentation

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Layers
    subgraph "SQL Client Layer"
        direction TB
        SQLClient["PL/pgSQL API & Migrations"]:::blue
    end

    subgraph "PostgreSQL Server"
        direction TB
        SQLAPI["SQL API (net.http_* functions)"]:::blue
        subgraph "pg_net Extension"
            direction TB
            Core["core.c/.h"]:::green
            Worker["BGWorker (worker.c)"]:::green
            Event["Event Loop (event.c/.h)"]:::green
            Util["Utilities (util.c/.h)"]:::green
            Errors["Error Handling (errors.c/.h)"]:::green
            CurlPrelude["libcurl Prelude (curl_prelude.h)"]:::green
            PGPrelude["Postgres Prelude (pg_prelude.h)"]:::green
        end
        subgraph "Unlogged Queues"
            direction TB
            RequestQueue["net.http_request_queue"]:::green
            ResponseTable["net._http_response"]:::green
        end
    end

    subgraph "External HTTP Services"
        direction TB
        External["Any REST API / Webhook"]:::orange
    end

    subgraph "Dev/CI Tooling"
        direction TB
        NixEnv["Nix Environment"]:::grey
        ShellNix["shell.nix"]:::grey
        CI["GitHub Actions CI"]:::grey
        DocsDir["docs/"]:::grey
        MkdocsCfg["mkdocs.yml"]:::grey
        MakefileNode["Makefile"]:::grey
        ClangFmt[".clang-format"]:::grey
    end

    %% Data Flows
    SQLClient -->|"CALL net.http_*"| SQLAPI
    SQLAPI -->|"INSERT request"| RequestQueue
    Worker -->|"poll queue"| RequestQueue
    Worker -->|"HTTP request via libcurl"| External
    External -->|"HTTP response"| Worker
    Worker -->|"INSERT/UPDATE response"| ResponseTable
    SQLClient -->|"SELECT results"| ResponseTable

    %% Click Events for component mapping
    click SQLClient "https://github.com/supabase/pg_net/tree/master/sql/"
    click Core "https://github.com/supabase/pg_net/blob/master/src/core.c"
    click Worker "https://github.com/supabase/pg_net/blob/master/src/worker.c"
    click Event "https://github.com/supabase/pg_net/blob/master/src/event.c"
    click Util "https://github.com/supabase/pg_net/blob/master/src/util.c"
    click Errors "https://github.com/supabase/pg_net/blob/master/src/errors.c"
    click CurlPrelude "https://github.com/supabase/pg_net/blob/master/src/curl_prelude.h"
    click PGPrelude "https://github.com/supabase/pg_net/blob/master/src/pg_prelude.h"
    click RequestQueue "https://github.com/supabase/pg_net/blob/master/sql/pg_net--*.sql"
    click ResponseTable "https://github.com/supabase/pg_net/blob/master/sql/pg_net--*.sql"
    click NixEnv "https://github.com/supabase/pg_net/tree/master/nix/"
    click ShellNix "https://github.com/supabase/pg_net/blob/master/shell.nix"
    click CI "https://github.com/supabase/pg_net/blob/master/.github/workflows/main.yml"
    click DocsDir "https://github.com/supabase/pg_net/tree/master/docs/"
    click MkdocsCfg "https://github.com/supabase/pg_net/blob/master/mkdocs.yml"
    click MakefileNode "https://github.com/supabase/pg_net/tree/master/Makefile"
    click ClangFmt "https://github.com/supabase/pg_net/blob/master/.clang-format"

    %% Styles
    classDef blue fill:#BBDEFB,stroke:#0D47A1,color:#0D47A1
    classDef green fill:#C8E6C9,stroke:#1B5E20,color:#1B5E20
    classDef orange fill:#FFE0B2,stroke:#E65100,color:#E65100
    classDef grey fill:#E0E0E0,stroke:#424242,color:#424242
```
