# trendyol/go-dcp-kafka architecture

> Architecture diagram of the GitHub repository trendyol/go-dcp-kafka, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/trendyol/go-dcp-kafka
- Repository: https://github.com/trendyol/go-dcp-kafka

## Overview

##### Main Components

a) Core Components:
- Couchbase Database (Source)
- DCP Protocol Handler
- Connector Core
- Kafka Producer
- Kafka Cluster (Destination)
- Metadata Storage (Couchbase/Kafka/File)

b) Supporting Components:
- Configuration Management
- Metrics Collection System
- Event Mapping System
- Batch Processing System

**┌─ Source Layer ─────┐  ┌─ Processing Layer ──────┐  ┌─ Destination Layer ─┐**

- Couchbase      │──▶│    DCP Protocol    ────┼──▶│    Kafka Cluster    │
- Database       │  │    Handler             │  │                      │

- ┘  │         ▼              │  └──────────────────────┘
- Connector Core        │

- ▼              │
- Event Mapper         │

- ▼              │
- Batch Processor      │

- ┘

##### Key Interactions to Highlight

- DCP event flow (Couchbase → DCP Handler)
- Document transformation flow (DCP Handler → Event Mapper)
- Batch processing flow (Event Mapper → Batch Processor)
- Kafka message production (Batch Processor → Kafka)

##### Important Subsystems

a) Configuration Management:
- How configuration flows to different components
- Different configuration options (Kafka, DCP, Metadata)

b) Batch Processing:
- Batch size limits
- Batch timing mechanism
- Compression options

c) Event Mapping:
- Demonstrate the transformation pipeline
- Collection-to-topic mapping

##### Monitoring & Metrics

A separate section showing:
- Prometheus metrics collection
- Grafana dashboard integration
- Key performance indicators
- Latency metrics
- Batch processing metrics
- DCP metrics

##### Additional Details

Callouts for:
- Security configurations (SSL/TLS, SCRAM)
- Scaling capabilities
- Error handling paths
- Checkpoint management

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Source
        CB[Couchbase Database]:::source
    end

    subgraph Processing
        DCP[DCP Protocol Handler]:::processing
        CC[Connector Core]:::processing
        EM[Event Mapper]:::processing
        BP[Batch Processor]:::processing
        
        subgraph Configuration
            CFG[Configuration Management]:::config
        end
        
        subgraph Monitoring
            MC[Metrics Collector]:::monitoring
            GD[Grafana Dashboard]:::monitoring
            PM[Prometheus Metrics]:::monitoring
        end
    end

    subgraph Destination
        KP[Kafka Producer]:::destination
        KC[Kafka Cluster]:::destination
        KM[Kafka Metadata]:::destination
    end

    %% Main Data Flow
    CB --> DCP
    DCP --> CC
    CC --> EM
    EM --> BP
    BP --> KP
    KP --> KC

    %% Configuration Flow
    CFG -.-> DCP
    CFG -.-> CC
    CFG -.-> KP
    
    %% Monitoring Flow
    DCP --o MC
    CC --o MC
    BP --o MC
    KP --o MC
    MC --> PM
    PM --> GD
    
    %% Metadata Flow
    KM <-.-> KP
    
    %% Styling
    classDef source fill:#3498db
    classDef processing fill:#2ecc71
    classDef destination fill:#e67e22
    classDef monitoring fill:#9b59b6
    classDef config fill:#95a5a6
    
    %% Click Events
    click DCP "https://github.com/Trendyol/go-dcp-kafka/blob/master/dcp_event_handler.go"
    click CC "https://github.com/Trendyol/go-dcp-kafka/blob/master/connector.go"
    click KP "https://github.com/Trendyol/go-dcp-kafka/tree/master/kafka/producer"
    click EM "https://github.com/Trendyol/go-dcp-kafka/blob/master/mapper.go"
    click CFG "https://github.com/Trendyol/go-dcp-kafka/blob/master/config/config.go"
    click MC "https://github.com/Trendyol/go-dcp-kafka/blob/master/metric/collector.go"
    click KM "https://github.com/Trendyol/go-dcp-kafka/blob/master/kafka/metadata/kafka_metadata.go"
    click GD "https://github.com/Trendyol/go-dcp-kafka/blob/master/example/grafana/grafana/provisioning/dashboards/dashboard.json"
    click PM "https://github.com/Trendyol/go-dcp-kafka/blob/master/example/grafana/prometheus/prometheus.yml"

    %% Legend
    subgraph Legend
        S[Source System]:::source
        P[Processing Component]:::processing
        D[Destination System]:::destination
        M[Monitoring Component]:::monitoring
        C[Configuration]:::config
        
        %% Legend Relationships
        L1["--> Data Flow"]
        L2["-.-> Configuration Flow"]
        L3["--o Metrics Flow"]
    end
```
