# meltanolabs/tap-postgres architecture

> Architecture diagram of the GitHub repository meltanolabs/tap-postgres, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/meltanolabs/tap-postgres
- Repository: https://github.com/meltanolabs/tap-postgres

## Overview

##### Main Components

a. Core Components:
- PostgreSQL Source Database
- Tap-Postgres Extractor
- Singer Message Output
- Configuration Management
- State Management

b. Special Features Components:
- SSL Connection Handler
- SSH Tunnel Manager
- Log-Based Replication Manager
- Stream Maps Processor
- Schema Flattening Engine

##### Component Relationships and Data Flow

**[Source Layer]**
**PostgreSQL Database (with optional wal2json plugin)**
- ↓
**[Security Layer]**
**SSL Handler ←→ SSH Tunnel Manager**
- ↓
**[Core Processing Layer]**
**Tap-Postgres Core**
- Client Handler (client.py)
- Tap Implementation (tap.py)
- ↓
**[Stream Processing Layer]**
- Stream Maps Processor
- Schema Flattening Engine
- ↓
**[Output Layer]**
**Singer Message Output (JSON)**

**┌→ Full Table Replication**
- → Incremental Replication
- → Log-Based Replication (with wal2json)

**[SSL Certificates]**
- Client Certs
- Server Certs
- CA Certs
- ↓
**[SSH Tunnel]**
- Bastion Server
- Key Management
- ↓
**[Database Connection]**

##### Implementation Details

Key files and their relationships:
**client.py ←→ tap.py**
- ↓
**Configuration Files**
- meltano.yml
- SSL Configs
- SSH Configs

##### Testing Infrastructure

A separate section showing:
**[Test Components]**
- Unit Tests
- Integration Tests
- Configuration Tests
- Resource Fixtures

##### Additional Elements

- Version compatibility information (Python 3.9-3.13, PostgreSQL 12-17)
- Configuration options flow
- Plugin system integration points

##### Metadata and Documentation

- Supported capabilities
- Configuration options
- Environmental variables support
- SDK integration points

This structure will provide a clear understanding of how the tap works while maintaining the technical depth needed for a principal software engineer to understand the system's architecture.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph SRC["Source Layer"]
        PG[("PostgreSQL Database")]
        WAL["WAL2JSON Plugin"]
        PG --> WAL
    end

    subgraph SEC["Security Layer"]
        SSL["SSL Connection Handler"]:::ssl
        SSH["SSH Tunnel Manager"]:::ssh
        SSL <--> SSH
    end

    subgraph CORE["Core Processing Layer"]
        TAP["Tap-Postgres Core"]:::tap
        CLIENT["Client Handler"]:::client
        TAP <--> CLIENT
    end

    subgraph STREAM["Stream Processing Layer"]
        SMP["Stream Maps Processor"]
        SFE["Schema Flattening Engine"]
    end

    subgraph OUT["Output Layer"]
        SINGER["Singer Message Output"]
    end

    subgraph REP["Replication Methods"]
        FULL["Full Table"]
        INCR["Incremental"]
        LOG["Log-Based"]
    end

    subgraph TEST["Testing Infrastructure"]
        UNIT["Unit Tests"]:::test
        INT["Integration Tests"]:::test
        CONF["Configuration Tests"]:::test
        FIX["Resource Fixtures"]:::test
    end

    subgraph CONFIG["Configuration"]
        MELT["Meltano Config"]:::config
        SSLC["SSL Config"]:::config
        SSHC["SSH Config"]:::config
    end

    %% Main Flow
    SRC --> SEC
    SEC --> CORE
    CORE --> STREAM
    STREAM --> OUT

    %% Replication Methods Connection
    CORE --> REP
    REP --> STREAM

    %% Configuration Connections
    CONFIG --> CORE
    CONFIG --> SEC

    %% Testing Connections
    TEST --> CORE
    TEST --> SEC

    %% Styling
    classDef ssl fill:#90EE90
    classDef ssh fill:#98FB98
    classDef tap fill:#87CEEB
    classDef client fill:#87CEFA
    classDef test fill:#D3D3D3
    classDef config fill:#FFB6C1

    %% Click Events
    click TAP "https://github.com/meltanolabs/tap-postgres/blob/main/tap_postgres/tap.py"
    click CLIENT "https://github.com/meltanolabs/tap-postgres/blob/main/tap_postgres/client.py"
    click SSL "https://github.com/meltanolabs/tap-postgres/tree/main/ssl"
    click SSH "https://github.com/meltanolabs/tap-postgres/tree/main/ssh_tunnel"
    click LOG "https://github.com/meltanolabs/tap-postgres/tree/main/log_based"
    click MELT "https://github.com/meltanolabs/tap-postgres/blob/main/meltano.yml"
    click UNIT "https://github.com/meltanolabs/tap-postgres/blob/main/tests/test_core.py"
    click INT "https://github.com/meltanolabs/tap-postgres/blob/main/tests/test_log_based.py"
    click CONF "https://github.com/meltanolabs/tap-postgres/blob/main/tests/test_ssl.py"
    click FIX "https://github.com/meltanolabs/tap-postgres/blob/main/tests/resources/data.json"

    %% Legend
    subgraph Legend
        L1["Core Components"]:::tap
        L2["Security Components"]:::ssl
        L3["Testing Components"]:::test
        L4["Configuration"]:::config
    end
```
