# supabase/etl architecture

> Architecture diagram of the GitHub repository supabase/etl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/supabase/etl
- Repository: https://github.com/supabase/etl

## Overview

##### Project Type and Purpose
- ETL is a Rust-based, full-stack framework and set of services for building real-time Postgres logical replication applications.
- It consists of multiple crates and services that work together:
- etl: core library and CLI for defining and running replication pipelines
- etl-api: HTTP management API for configuring pipelines, destinations, and tenants
- etl-replicator: long-running service (or K8s job) that streams WAL changes and delivers them to destinations
- etl-postgres: Postgres replication client abstraction
- etl-destinations: pluggable destination connectors (BigQuery, Iceberg, etc.)
- etl-config: unified configuration loader
- etl-telemetry: metrics & tracing
- etl-examples & etl-benchmarks: sample apps & performance tests

##### File Structure Analysis
- Top-level workspace Cargo.toml ties together all sub-crates under an ETL umbrella.
- Directories:
- etl-api: HTTP REST service with its own DB migrations, configuration files, Dockerfile, and K8s code generation
- etl-replicator: standalone service that reads config, connects to Postgres, streams changes, and writes to destinations
- etl-postgres: library that encapsulates logical replication protocol interactions
- etl-destinations: modular connectors for various sinks
- etl-config & etl-telemetry: shared support libraries
- scripts/: Docker Compose and K8s YAML for spinning up a full ETL data plane
- docs/explanation/architecture.md already contains some guidance
- CI/CD pipelines and GitHub Actions show Docker build, audit, docs generation

##### Key README Insights
- ETL sits “on top of Postgres logical replication” to stream changes
- Supports batching, parallelism, fault tolerance, custom destinations
- Examples show in-memory pipeline, but production uses etl-replicator + etl-api

##### Components & Interactions
a. Main Components
- Client / Management Plane (etl-api)
- Control Database (Postgres metadata DB)
- Replicator Service (etl-replicator pods or processes)
- Core Library (etl + etl-postgres + etl-config + etl-telemetry)
- Destination Connectors (BigQuery, Iceberg, custom via etl-destinations)
- State/Schema Store (Postgres or in-memory via etl-store)
- External Services: Kubernetes API Server, Object Storage (optional), Monitoring/Tracing

b. Relationships & Data Flow
1. Administrator uses REST API (etl-api) to create tenants, sources, publications, pipelines, and destinations.
2. etl-api writes configuration into the Control DB (tables: tenants, sources, pipelines, destinations).
3. etl-api (optionally) calls Kubernetes API to launch etl-replicator pods with mounted config.
4. etl-replicator reads its configuration (from YAML or DB), connects to Postgres publication via logical replication slot.
5. etl-postgres client streams WAL changes into Rust types (ChangeRecords).
6. Core pipeline module (etl crate) applies conversion, batching, parallel dispatch to workers.
7. Workers invoke destination connectors (etl-destinations) to push data to sinks (BigQuery, Iceberg).
8. State and schema metadata are persisted back to State Store (Postgres or configured store backend).
9. etl-telemetry emits metrics/traces to observability backends.

c. Architectural Patterns & Principles
- Microservice separation: Management Plane vs. Data Plane
- Pub/Sub replication: Postgres as source, destinations as subscribers
- Modular plugin architecture: destinations and stores are traits implemented by crates
- Concurrency and batching: worker pool, configurable batch sizes/timings
- Fault tolerance: retry logic, idempotent writes, state persistence

d. Key Technologies & Libraries
- Rust async runtime: Tokio
- SQLx for Postgres queries & migrations (etl-api, etl-postgres)
- Kubernetes API client (in etl-api and etl-replicator k8s module)
- serde for config serialization/deserialization
- tracing & metrics crates in etl-telemetry

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Layer
    subgraph "User & Client Layer"
        A["User / Administrator"]:::external
    end

    %% Management Plane
    subgraph "Management Plane" 
        B["REST API (etl-api)"]:::core
    end

    %% Persistence Layer
    subgraph "Persistence Layer"
        C["Control DB (Postgres)"]:::db
        D["State Store (Postgres or in-memory)"]:::db
    end

    %% Data Plane
    subgraph "Data Plane"
        E["Replicator Service\n(etl-replicator)"]:::core
        E -->|"reads config"| C
        E -->|"Stream changes"| F[("Postgres WAL Slot")]:::db
    end

    %% Core Library & Plugins
    subgraph "Core & Plugin Libraries"
        G["Core Pipeline (etl)"]:::lib
        H["Postgres Client (etl-postgres)"]:::lib
        I["Config Loader (etl-config)"]:::lib
        J["Telemetry (etl-telemetry)"]:::lib
        K["Destination Connectors\n(etl-destinations)"]:::lib
    end

    %% Destination Layer
    subgraph "Destination Layer"
        L["BigQuery"]:::external
        M["Iceberg"]:::external
        N["Custom Connector"]:::external
    end

    %% Observability
    subgraph "Observability"
        O["Monitoring / Tracing"]:::external
    end

    %% Scripts & Examples
    subgraph "Examples & Scripts"
        P["Example Apps (etl-examples)"]:::lib
        Q["Benchmarks (etl-benchmarks)"]:::lib
        R["Orchestration Scripts (scripts/)"]:::external
    end

    %% Data Flows
    A -->|"calls REST API"| B
    B -->|"Create pipeline"| C
    B -->|"Invoke K8s API"| S[Kubernetes API Server]:::external
    S -->|"Launch pods"| E

    E -->|"uses Postgres client"| H
    E -->|"invokes pipeline"| G
    E -->|"loads config"| I
    E -->|"emits metrics"| J
    G -->|"uses connectors"| K
    K -->|"Batch write"| L
    K -->|"Batch write"| M
    K -->|"Batch write"| N
    G -->|"persist metadata"| D
    J -->|"send traces"| O

    P -->|"in-memory pipeline"| G
    Q -->|"performance tests"| G
    R -.-> B
    R -.-> E

    %% Click Events
    click B "https://github.com/supabase/etl/tree/main/etl-api/"
    click E "https://github.com/supabase/etl/tree/main/etl-replicator/"
    click H "https://github.com/supabase/etl/tree/main/etl-postgres/"
    click K "https://github.com/supabase/etl/tree/main/etl-destinations/"
    click I "https://github.com/supabase/etl/tree/main/etl-config/"
    click J "https://github.com/supabase/etl/tree/main/etl-telemetry/"
    click G "https://github.com/supabase/etl/tree/main/etl/"
    click P "https://github.com/supabase/etl/tree/main/etl-examples/"
    click Q "https://github.com/supabase/etl/tree/main/etl-benchmarks/"
    click R "https://github.com/supabase/etl/tree/main/scripts/"

    %% Styles
    classDef core fill:#dae8fc,stroke:#6c8ebf,color:#000
    classDef db fill:#d5e8d4,stroke:#82b366,color:#000
    classDef lib fill:#f5f5f5,stroke:#999,color:#000
    classDef external fill:#ffe6cc,stroke:#d79b00,color:#000
```
