# peerdb-io/peerdb architecture

> Architecture diagram of the GitHub repository peerdb-io/peerdb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/peerdb-io/peerdb
- Repository: https://github.com/peerdb-io/peerdb

## Overview

##### Main Components
a. ETL Engine (“Flow” Subsystem)
- The “flow” directory is the heart of the streaming logic. It holds the core engine, connectors for various targets (e.g., BigQuery, ClickHouse, Kafka, Postgres, etc.), and the orchestration code (found in subdirectories like workflows, cmd, internal, and connectors).
- Within this subsystem, highlight the workflow orchestration elements from flow/workflows – these coordinate the steps for change data capture, initial load, maintenance flows, and scheduled synchronizations. Emphasize that these workflows handle orchestration, state management, retries, and consistency checks.
b. Metadata and Catalog (“Nexus” Subsystem)
- The “nexus” directory (built in Rust) includes components for catalog migrations, managing peer connections, and query parsing. This service likely stores configuration and metadata necessary for dynamic workflow operations.
c. User Interface (UI)
- The “ui” directory embodies the web application for monitoring, administration, and configuration. It enables users to launch, observe, and troubleshoot ETL workflows.
d. Infrastructure and Deployment:
- Docker-compose files, Dockerfiles in stacks, and CI/CD workflows show the containerized deployment of databases (Postgres as a catalog, and MinIO for staging), the ETL engine, and supporting services.
- Mention the integration with Temporal (or another workflow orchestrator) for fault-tolerant, stateful workflow management – as noted in the README and internal configuration files.
e. External Dependencies:
- Postgres is the primary data source.
- Target systems include data warehouses (BigQuery, ClickHouse, Snowflake), message queues (Kafka, PubSub, EventHub), and storage (S3/MinIO).
- Services like MinIO (for staging large TOAST columns) and external applications (e.g., ClickHouse Cloud) are also part of the ecosystem.

##### Relationships and Data Flow
– from Postgres to the Flow API and Workers,
– then through data transformation layers and connectors,
– and finally landing in target systems.
- How metadata flows from the Nexus catalog service into the ETL engine, thereby guiding workflows with schema and configuration data.
- Staging process via MinIO, where data is temporarily stored before being loaded into targets, especially when targets like ClickHouse require external access.

##### Architectural Patterns
- Microservices/Modular Design: The separation into “flow”, “nexus”, and “ui” emphasizes modularity.
- Workflow Orchestration: The “workflows” inside the flow engine indicate a clear separation between orchestration logic and connector logic.
- Containerized Deployment: The use of Dockerfiles, docker-compose, and CI/CD workflows point to container-based deployment orchestrated via scripts and Kubernetes (or similar) in production.
- Extensibility: The wide variety of connectors (organized within the connectors subdirectories) demonstrates a plugin-like architecture. This makes it easy to add or modify streaming pipelines.
- Fault Tolerance and Performance: Furthermore, the architecture includes automatic retries, state management, and parallel worker processes that ensure robustness and efficiency.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Main Components
    UI["UI & API Gateway"]:::ui
    Postgres["Postgres (Source)"]:::datasource
    Nexus["Metadata & Catalog (Nexus)"]:::metadata

    subgraph "ETL Engine (Flow)"
        FlowCore["Flow Core & Connectors"]:::etlCore
        Workflow["Workflow Orchestration"]:::workflow
    end

    Targets["Target Connectors (Data Warehouses,Queues,Storage)"]:::targets
    Minio["MinIO Staging"]:::staging
    Orchestration["Orchestration Service (Temporal)"]:::orchestration

    subgraph "Infrastructure & Deployment"
        Deploy1["Deploy: docker-compose.yml"]:::infra
        Deploy2["Deploy: docker-compose-dev.yml"]:::infra
        Deploy3["Deploy: stacks/"]:::infra
        Deploy4["Deploy: .github/workflows/"]:::infra
    end

    DC["Deployment Config"]:::infra

    %% Relationships and Data Flow
    UI -->|"trigger"| Workflow
    UI -->|"monitor"| FlowCore
    Workflow -->|"orchestrates"| FlowCore
    Postgres -->|"streams"| FlowCore
    Nexus -->|"provides_config"| FlowCore
    UI -->|"manages"| Nexus
    FlowCore -->|"loads"| Targets
    FlowCore -->|"stages"| Minio
    Workflow -->|"submits_tasks"| Orchestration

    %% Deployment Integration
    Deploy1 --> DC
    Deploy2 --> DC
    Deploy3 --> DC
    Deploy4 --> DC
    DC ---|"deploys"| UI
    DC ---|"deploys"| Workflow
    DC ---|"deploys"| Nexus

    %% Click Events
    click FlowCore "https://github.com/peerdb-io/peerdb/tree/main/flow/"
    click Workflow "https://github.com/peerdb-io/peerdb/tree/main/flow/workflows"
    click Nexus "https://github.com/peerdb-io/peerdb/tree/main/nexus/"
    click UI "https://github.com/peerdb-io/peerdb/tree/main/ui/"
    click Deploy1 "https://github.com/peerdb-io/peerdb/blob/main/docker-compose.yml"
    click Deploy2 "https://github.com/peerdb-io/peerdb/blob/main/docker-compose-dev.yml"
    click Deploy3 "https://github.com/peerdb-io/peerdb/tree/main/stacks/"
    click Deploy4 "https://github.com/peerdb-io/peerdb/tree/main/.github/workflows/"

    %% Styles
    classDef ui fill:#FFF4E6,stroke:#000,stroke-width:2px;
    classDef datasource fill:#FFE6E6,stroke:#000,stroke-width:2px;
    classDef metadata fill:#E6FFE6,stroke:#000,stroke-width:2px;
    classDef etlCore fill:#E6F2FF,stroke:#000,stroke-width:2px;
    classDef workflow fill:#FFD699,stroke:#000,stroke-width:2px;
    classDef targets fill:#F2F2F2,stroke:#000,stroke-width:2px;
    classDef staging fill:#E0F7FA,stroke:#000,stroke-width:2px;
    classDef orchestration fill:#FFF9C4,stroke:#000,stroke-width:2px;
    classDef infra fill:#F0E6FF,stroke:#000,stroke-width:2px;
```
