How supabase/pg_net works
Project Type and Purpose
• This is a PostgreSQL extension (“pg_net”) written in C that adds asynchronous HTTP/HTTPS capabilities to the database engine.
• It integrates deeply with PostgreSQL via a background worker process, unlogged tables for queuing, and SQL-callable functions.
File Structure Analysis
• src/ contains the C source files and headers (core.c, worker.c, event.c, util.c, errors.c, plus CURL and PG Prelude headers).
• pg_net.control.in and sql/ contain extension metadata and versioned SQL migrations for table creation and upgrade paths.
• test/ holds Python-based integration tests (conftest.py, test_engine.py, HTTP behavior tests, worker error tests).
• nix/ and shell.nix define the reproducible dev environment (Nix build of libcurl, PostgreSQL, Nginx config for docs).
• docs/ and mkdocs.yml drive the published documentation.
• Makefile & GitHub Actions workflow for build, test, coverage, and publish.
Key Components
a. SQL Client Layer
• SQL scripts, psql clients, Trigger functions (PL/pgSQL) and Cron jobs (PG_CRON) that call net.http_get/post/delete.
b. PostgreSQL Server
• Core Postgres process with shared_preload_libraries = 'pg_net'.
• The pg_net background worker thread/process registered via the PostgreSQL BGWorker API.
c. pg_net Extension Modules (inside PostgreSQL process)
• Producer: SQL-callable functions in PL/pgSQL or C stubs that INSERT into net.http_request_queue.
• Consumer: Background worker polling net.http_request_queue, executing requests via libcurl, and writing to net._http_response.
• Internal C components:
• core.c – extension init and function exports
• worker.c – background worker loop & scheduling
• event.c – event-loop integration
• util.c + errors.c – helper routines
d. Persistence Queues (Unlogged Tables)
• net.http_request_queue (pending requests)
• net._http_response (completed results and errors)
e. External HTTP/HTTPS Services
• Any REST API endpoints called asynchronously (webhooks, serverless functions, data sync targets).
f. Dev/CI Tooling (side components)
• Nix environment for reproducible builds
• GitHub Actions for compile/test/coverage
• MkDocs for documentation site
Relationships and Data Flows
• [SQL Client / Trigger / Cron] --(SQL CALL net.http_*)-> [http_request_queue]
• [pg_net Background Worker] --(poll queue)-> [http_request_queue]
• [Background Worker] --(libcurl HTTP request)-> [External HTTP Service]
• [External HTTP Service] --(HTTP response)-> [Background Worker]
• [Background Worker] --(INSERT/UPDATE response)-> [net._http_response]
• [Monitoring SQL Queries] --(SELECT)-> [net._http_response]
Architectural Patterns & Principles
• Producer–Consumer via database queue.
• Background worker (asynchronous processing) integrated into the database.
• Unlogged tables for high-performance queuing.
• Modular C design separating core, worker, events, utilities, and error handling.
• PL/pgSQL wrappers for user-friendly SQL API.
Technologies, Frameworks, Libraries
• PostgreSQL BGWorker API
• libcurl for HTTP/HTTPS
• C (ISO C) and PGXS build system (Makefile)
• PL/pgSQL for extension functions
• Python + pytest for integration tests
• Nix for reproducible dev/deployment
• GitHub Actions CI
• MkDocs for documentation