# sequinstream/sequin architecture

> Architecture diagram of the GitHub repository sequinstream/sequin, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sequinstream/sequin
- Repository: https://github.com/sequinstream/sequin

## Overview

##### Main Components Overview
The system should be divided into these major components:

a) Core Components:
- PostgreSQL Connection Layer
- Change Data Capture Engine
- Message Processing Pipeline
- Sink Management System
- Web Interface
- Authentication & Authorization

b) Sink Integrations:
- Kafka
- SQS
- Redis
- HTTP Webhooks
- GCP Pub/Sub
- NATS
- RabbitMQ

##### Detailed Component Breakdown

Frontend Layer:
- Svelte-based UI (/assets/svelte)
- LiveView Integration
- Real-time Updates
- Configuration Interface

Backend Core:
- Elixir/Phoenix Application
- Database Management (/lib/sequin/databases)
- Consumer Management (/lib/sequin/consumers)
- Replication Management (/lib/sequin/replication)
- Health Monitoring (/lib/sequin/health)

Integration Layer:
- Sink Adapters (/lib/sequin/sinks)
- Message Transformation
- Error Handling
- Retry Logic

##### Data Flow Representation

Following flows:
**PostgreSQL Source → CDC Engine → Message Processing → Sink Delivery**
- ↓
- Message Storage
- ↓
- Retry/Error Handling

##### Key Design Elements

Use these visual elements:

##### Component Relationships

Interactions:
- PostgreSQL logical replication slots
- Message transformation pipeline
- Sink delivery mechanisms
- Health monitoring system
- Authentication flow

##### Specific Implementation Details

Technical aspects:
- Elixir/Phoenix backend structure
- LiveView + Svelte frontend architecture
- Database connections and pooling
- Message queue management
- Sink-specific protocols

##### Monitoring & Health Checks

Systems:
- Health monitoring (/lib/sequin/health)
- Metrics collection (/lib/sequin/metrics)
- Logging system (/lib/sequin/logs)
- Error tracking (/lib/sequin/error)

##### Real-time monitoring capabilities

Use clear visual hierarchy:
- Top level: Main system components
- Middle level: Internal services
- Bottom level: External integrations

- State indicators
- Health status representations

This will create a comprehensive visualization that accurately represents Sequin's architecture while maintaining clarity and usefulness for technical discussions.

Remember to maintain appropriate abstraction levels while showing critical implementation details that make Sequin unique in the CDC space.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles and classes
    classDef core fill:#2b6cb0,stroke:#1a4971,color:white
    classDef frontend fill:#805ad5,stroke:#553c9a,color:white
    classDef sink fill:#38a169,stroke:#2f855a,color:white
    classDef monitoring fill:#d69e2e,stroke:#b7791f,color:white
    classDef error fill:#e53e3e,stroke:#c53030,color:white
    classDef storage fill:#718096,stroke:#4a5568,color:white

    %% Core Components
    subgraph CoreSystem
        PG[PostgreSQL Connection Layer]:::core
        CDC[Change Data Capture Engine]:::core
        MsgProc[Message Processing Pipeline]:::core
        SinkMgmt[Sink Management System]:::core
    end

    %% Frontend Components
    subgraph FrontendSystem
        WebUI[Web Interface]:::frontend
        Auth[Authentication & Authorization]:::frontend
    end

    %% Monitoring Components
    subgraph MonitoringSystem
        Health[Health Monitoring]:::monitoring
        Metrics[Metrics Collection]:::monitoring
        Logs[Logging System]:::monitoring
    end

    %% Sink Integrations
    subgraph SinkIntegrations
        Kafka[Kafka]:::sink
        Redis[Redis]:::sink
        HTTP[HTTP Webhooks]:::sink
        GCP[GCP Pub/Sub]:::sink
        NATS[NATS]:::sink
        RMQ[RabbitMQ]:::sink
    end

    %% Data Flow
    PG --> CDC
    CDC --> MsgProc
    MsgProc --> SinkMgmt
    SinkMgmt --> |Success| SinkIntegrations
    SinkMgmt --> |Error| MsgProc

    %% Monitoring Flow
    CoreSystem --> Health
    CoreSystem --> Metrics
    CoreSystem --> Logs

    %% Frontend Interactions
    WebUI --> Auth
    Auth --> CoreSystem

    %% Click Events
    click PG "https://github.com/sequinstream/sequin/tree/main/lib/sequin/databases"
    click CDC "https://github.com/sequinstream/sequin/tree/main/lib/sequin/databases_runtime"
    click MsgProc "https://github.com/sequinstream/sequin/tree/main/lib/sequin/consumers_runtime"
    click SinkMgmt "https://github.com/sequinstream/sequin/tree/main/lib/sequin/sinks"
    click WebUI "https://github.com/sequinstream/sequin/tree/main/assets/svelte"
    click Auth "https://github.com/sequinstream/sequin/tree/main/lib/sequin/accounts"
    click Kafka "https://github.com/sequinstream/sequin/tree/main/lib/sequin/sinks/kafka"
    click Redis "https://github.com/sequinstream/sequin/tree/main/lib/sequin/sinks/redis"
    click HTTP "https://github.com/sequinstream/sequin/blob/main/lib/sequin_web/controllers/push_webhook_controller.ex"
    click GCP "https://github.com/sequinstream/sequin/tree/main/lib/sequin/sinks/gcp"
    click NATS "https://github.com/sequinstream/sequin/tree/main/lib/sequin/sinks/nats"
    click RMQ "https://github.com/sequinstream/sequin/tree/main/lib/sequin/sinks/rabbitmq"
    click Health "https://github.com/sequinstream/sequin/tree/main/lib/sequin/health"
    click Metrics "https://github.com/sequinstream/sequin/tree/main/lib/sequin/metrics"
    click Logs "https://github.com/sequinstream/sequin/tree/main/lib/sequin/logs"

    %% Legend
    subgraph Legend
        L1[Core Components]:::core
        L2[Frontend Components]:::frontend
        L3[Sink Integrations]:::sink
        L4[Monitoring]:::monitoring
    end
```
