How supabase/etl works
Project Type and Purpose
• ETL is a Rust-based, full-stack framework and set of services for building real-time Postgres logical replication applications.
• It consists of multiple crates and services that work together:
• etl: core library and CLI for defining and running replication pipelines
• etl-api: HTTP management API for configuring pipelines, destinations, and tenants
• etl-replicator: long-running service (or K8s job) that streams WAL changes and delivers them to destinations
• etl-postgres: Postgres replication client abstraction
• etl-destinations: pluggable destination connectors (BigQuery, Iceberg, etc.)
• etl-config: unified configuration loader
• etl-telemetry: metrics & tracing
• etl-examples & etl-benchmarks: sample apps & performance tests
File Structure Analysis
• Top-level workspace Cargo.toml ties together all sub-crates under an ETL umbrella.
• Directories:
• etl-api: HTTP REST service with its own DB migrations, configuration files, Dockerfile, and K8s code generation
• etl-replicator: standalone service that reads config, connects to Postgres, streams changes, and writes to destinations
• etl-postgres: library that encapsulates logical replication protocol interactions
• etl-destinations: modular connectors for various sinks
• etl-config & etl-telemetry: shared support libraries
• scripts/: Docker Compose and K8s YAML for spinning up a full ETL data plane
• docs/explanation/architecture.md already contains some guidance
• CI/CD pipelines and GitHub Actions show Docker build, audit, docs generation
Key README Insights
• ETL sits “on top of Postgres logical replication” to stream changes
• Supports batching, parallelism, fault tolerance, custom destinations
• Examples show in-memory pipeline, but production uses etl-replicator + etl-api
Components & Interactions
a. Main Components
• Client / Management Plane (etl-api)
• Control Database (Postgres metadata DB)
• Replicator Service (etl-replicator pods or processes)
• Core Library (etl + etl-postgres + etl-config + etl-telemetry)
• Destination Connectors (BigQuery, Iceberg, custom via etl-destinations)
• State/Schema Store (Postgres or in-memory via etl-store)
• External Services: Kubernetes API Server, Object Storage (optional), Monitoring/Tracing
b. Relationships & Data Flow
1. Administrator uses REST API (etl-api) to create tenants, sources, publications, pipelines, and destinations.
2. etl-api writes configuration into the Control DB (tables: tenants, sources, pipelines, destinations).
3. etl-api (optionally) calls Kubernetes API to launch etl-replicator pods with mounted config.
4. etl-replicator reads its configuration (from YAML or DB), connects to Postgres publication via logical replication slot.
5. etl-postgres client streams WAL changes into Rust types (ChangeRecords).
6. Core pipeline module (etl crate) applies conversion, batching, parallel dispatch to workers.
7. Workers invoke destination connectors (etl-destinations) to push data to sinks (BigQuery, Iceberg).
8. State and schema metadata are persisted back to State Store (Postgres or configured store backend).
9. etl-telemetry emits metrics/traces to observability backends.
c. Architectural Patterns & Principles
• Microservice separation: Management Plane vs. Data Plane
• Pub/Sub replication: Postgres as source, destinations as subscribers
• Modular plugin architecture: destinations and stores are traits implemented by crates
• Concurrency and batching: worker pool, configurable batch sizes/timings
• Fault tolerance: retry logic, idempotent writes, state persistence
d. Key Technologies & Libraries
• Rust async runtime: Tokio
• SQLx for Postgres queries & migrations (etl-api, etl-postgres)
• Kubernetes API client (in etl-api and etl-replicator k8s module)
• serde for config serialization/deserialization
• tracing & metrics crates in etl-telemetry