# trendyol/go-pq-cdc architecture

> Architecture diagram of the GitHub repository trendyol/go-pq-cdc, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/trendyol/go-pq-cdc
- Repository: https://github.com/trendyol/go-pq-cdc

## Overview

##### Main Components

a. Core Components:
- PostgreSQL Database (Source)
- go-pq-cdc Connector
- Downstream Systems (Kafka, Elasticsearch, etc.)
- Monitoring Stack (Prometheus/Grafana)

b. Internal Components of go-pq-cdc:
- Configuration Manager
- Replication Manager
- Publication Manager
- Message Handler
- Metrics Collector
- HTTP Server

**[Source Layer]**
- PostgreSQL Database
- ↓
**[CDC Layer (go-pq-cdc)]**
- Connector Core
- ↓
**[Destination Layer]**
- Downstream Systems

**[Configuration Manager] → [Connector Core] ← [HTTP Server]**
- ↑                     ↓
- Config Files        [Replication Manager]
- ↓
- [Publication Manager]
- ↓
- [Message Handler]
- ↓
- [Metrics Collector]

##### Key Interactions to Highlight

a. Replication Flow:
- PostgreSQL WAL → Replication Slot → Message Handler
- Message Format Processing (Insert/Update/Delete)
- Message Delivery to Handler Function

b. Management Flow:
- Configuration Loading
- Metrics Collection
- Health Checking
- Slot Activity Monitoring

##### Important Technical Details

- Replication Slot Management
- Publication Configuration
- Message Format Types
- Metric Endpoints
- Health Check API

##### Specific Implementation Notes

- The passive/active availability model
- Logical replication capabilities
- Metric collection points
- Configuration injection points
- Error handling paths

##### Component Descriptions

- Component roles and responsibilities
- Key configuration parameters
- Monitoring points
- Integration interfaces

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Source
        PG[PostgreSQL Database]
    end

    subgraph CDC["CDC Layer (go-pq-cdc)"]
        subgraph Core
            CM["Configuration Manager"]:::config
            RM["Replication Manager"]:::replication
            PM["Publication Manager"]:::publication
            MH["Message Handler"]:::message
            MC["Metrics Collector"]:::metrics
            HS["HTTP Server"]:::http
            CONN["Connection Manager"]:::conn
            LOG["Logger"]:::log
            SLOT["Slot Manager"]:::slot
        end

        subgraph MessageProcessors["Message Processors"]
            INS["Insert Processor"]:::format
            UPD["Update Processor"]:::format
            DEL["Delete Processor"]:::format
            REL["Relation Handler"]:::format
        end
    end

    subgraph Destination
        DS1[Kafka]
        DS2[Elasticsearch]
        DS3["Other Systems"]
    end

    subgraph Monitoring
        PROM[Prometheus]
        GRAF[Grafana]
    end

    %% Core Flow
    PG --> CONN
    CONN --> RM
    RM --> PM
    PM --> MH
    MH --> |Format Messages| MessageProcessors
    MessageProcessors --> DS1
    MessageProcessors --> DS2
    MessageProcessors --> DS3

    %% Configuration and Management
    CM --> RM
    CM --> PM
    CM --> SLOT
    SLOT --> RM
    HS --> MC
    MC --> PROM
    PROM --> GRAF

    %% Logging
    LOG --> RM
    LOG --> PM
    LOG --> MH

    %% Click Events
    click CM "https://github.com/Trendyol/go-pq-cdc/tree/main/config/"
    click RM "https://github.com/Trendyol/go-pq-cdc/tree/main/pq/replication/"
    click PM "https://github.com/Trendyol/go-pq-cdc/tree/main/pq/publication/"
    click MH "https://github.com/Trendyol/go-pq-cdc/tree/main/pq/message/"
    click MC "https://github.com/Trendyol/go-pq-cdc/tree/main/internal/metric/"
    click HS "https://github.com/Trendyol/go-pq-cdc/blob/main/internal/http/server.go"
    click SLOT "https://github.com/Trendyol/go-pq-cdc/tree/main/pq/slot/"
    click CONN "https://github.com/Trendyol/go-pq-cdc/blob/main/pq/connection.go"
    click LOG "https://github.com/Trendyol/go-pq-cdc/blob/main/logger/logger.go"
    click INS "https://github.com/Trendyol/go-pq-cdc/blob/main/pq/message/format/insert.go"
    click UPD "https://github.com/Trendyol/go-pq-cdc/blob/main/pq/message/format/update.go"
    click DEL "https://github.com/Trendyol/go-pq-cdc/blob/main/pq/message/format/delete.go"
    click REL "https://github.com/Trendyol/go-pq-cdc/blob/main/pq/message/format/relation.go"

    %% Styles
    classDef config fill:#f9f,stroke:#333
    classDef replication fill:#bbf,stroke:#333
    classDef publication fill:#bfb,stroke:#333
    classDef message fill:#fbf,stroke:#333
    classDef metrics fill:#ff9,stroke:#333
    classDef http fill:#f99,stroke:#333
    classDef conn fill:#9ff,stroke:#333
    classDef log fill:#999,stroke:#333
    classDef slot fill:#f9b,stroke:#333
    classDef format fill:#bff,stroke:#333
    classDef external fill:#ddd,stroke:#333

    %% Legend
    subgraph Legend
        L1[Configuration]:::config
        L2[Replication]:::replication
        L3[Publication]:::publication
        L4[Message Handling]:::message
        L5[Metrics]:::metrics
        L6[HTTP Server]:::http
        L7[Connection]:::conn
        L8[Logging]:::log
        L9[Slot Management]:::slot
        L10[Message Format]:::format
        L11[External Systems]:::external
    end
```
