# trendyol/go-dcp-elasticsearch architecture

> Architecture diagram of the GitHub repository trendyol/go-dcp-elasticsearch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/trendyol/go-dcp-elasticsearch
- Repository: https://github.com/trendyol/go-dcp-elasticsearch

## Overview

##### Project Type and Purpose
- This is a Go-based connector designed to integrate Couchbase (as the source of change events) with Elasticsearch (the destination for indexing documents).
- The README and file structure indicate high performance and customizable mapping as key features. It supports batching, custom routing, and synchronization between the two systems.

##### File Structure
- Top-Level Components:
- Configuration: The “config” directory contains go files for configuration settings.
- Connector Logic: Files like connector.go and mapper.go indicate central processing logic to manage DCP events and map them into Elasticsearch actions.
- Elasticsearch-specific modules: Organized under the “elasticsearch” directory, including subdirectories for “client”, “document”, and “bulk” processing.
- Couchbase-related files: Located under the “couchbase” directory.
- Examples: The “example” directory contains various subprojects (e.g., default-mapper, doc-update, grafana) that illustrate different usage scenarios.
- Testing and Benchmarking: The “test” and “benchmark” directories highlight integration tests and performance benchmarking.
- CI/CD Integration: The .github directory with workflows shows continuous integration, testing, and release pipelines.

##### README for Additional Insights
- The README describes core functionality such as processing DCP events (e.g., mutation, expiration, deletion) to perform index or delete operations in Elasticsearch.
- It points out performance advantages, detailed configuration options (batch size, compression, etc.), and exposes metrics for observability.
- References to external tools like Grafana and Prometheus for monitoring suggest the importance of observability within the system.

a. Main Components:
- External Systems:
- Couchbase – Source of DCP events.
- Elasticsearch – Destination for document indexing.
- Monitoring Systems – Prometheus and Grafana for collecting metrics and visualizing performance.
- Internal Components:
- DCP Connector: Listens to and streams events from Couchbase.
- Mapper Module: Transforms Couchbase events into Elasticsearch actions/documents.
- Elasticsearch Client: Handles bulk requests, manages connection pooling, and processes indexing operations.
- Configuration Manager: Reads and applies configurations (from config files, environment variables, etc.).
- Metrics & Logging Modules: Capture metrics like request latency and action success/error counts; also includes handling of rejection logs.
- Batch Processor: Manages batching of events (by message count, size, and durations) before sending to Elasticsearch.
- CI/CD Infrastructure: Represented by GitHub workflows and Dockerfiles, illustrating how the system is built, tested, and deployed.

b. Relationships and Interactions:
- Data Flow:
1. Couchbase emits DCP events.
2. The DCP Connector receives these events and routes them to the Mapper.
3. The Mapper converts events to Elasticsearch action documents.
4. The Batch Processor aggregates these actions based on configured limits.
5. The Elasticsearch Client sends batched documents to Elasticsearch.
6. Metrics and logs are collected throughout the process and pushed to Prometheus/Grafana.
- Control Flow:
- Configuration settings (e.g., batch sizes, connection parameters) are applied in the Connector and Elasticsearch Client.
- Error handling (such as dealing with rejections) is processed via dedicated handlers.

c. Architectural Patterns and Design Principles:
- Modular Design: Each major function (data ingestion, mapping, batching, indexing) is handled by distinct modules.
- Real-Time Streaming: Designed for near real‑time processing of data changes.
- Batch Processing: Optimization of throughput and resource usage by aggregating events.
- Extensibility: Custom mapper functions (shown in various examples) allow for tailored transformations.
- Resilience and Observability: Metrics collection and logging (including rejection logs) ensure the system is robust and manageable.

d. Relevant Technologies and Frameworks:
- Go Programming Language with its standard library and concurrency model.
- Docker for containerization (as seen in multiple Dockerfiles and docker-compose.yml).
- GitHub Actions for CI/CD pipelines.
- Elasticsearch – utilized via custom clients and bulk APIs.
- Prometheus and Grafana – for monitoring and visualization of performance metrics.

- Use clear, labeled components for:
- Couchbase (DCP source)
- DCP Connector Module
- Mapper Module for event transformation
- Batch Processor managing request accumulation
- Elasticsearch Client handling document indexing
- Configuration Manager handling various configuration parameters
- Monitoring Components for metrics collection
- Data flow from Couchbase → Connector → Mapper → Batch Processor → Elasticsearch.
- Metrics/logs flowing to Monitoring tools.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Systems
    CB["((Couchbase (DCP Source)))"]:::external
    ES["((Elasticsearch))"]:::external
    MON["\"Prometheus & Grafana\""]:::external

    %% Internal Components
    CONFIG["\"Configuration Manager\""]:::internal
    DCP["\"DCP Connector Module\""]:::internal
    MAPPER["\"Mapper Module\""]:::internal
    BATCH["\"Batch Processor\""]:::internal
    ES_CLIENT["\"Elasticsearch Client\""]:::internal
    ES_DOC["\"Elasticsearch Document Handler\""]:::internal
    METRIC["\"Metrics Collection Module\""]:::internal
    CICD["\"CI/CD Infrastructure\""]:::cicd

    %% Subgraph for Couchbase Event Processing
    subgraph "Couchbase Event Processing"
        CE1["\"CB Event Reader\""]:::internal
        CE2["\"DCP Event Handler\""]:::internal
    end

    %% Subgraph for Rejection Log Handling
    subgraph "Rejection Log Handling"
        RL1["\"Rejection Log Sink Response Handler\""]:::internal
        RL2["\"Sink Response Handler\""]:::internal
    end

    %% Data Flow
    CB --> CE1
    CE1 --> CE2
    CE2 --> DCP
    CONFIG --> DCP
    DCP --> MAPPER
    MAPPER --> BATCH
    BATCH --> ES_CLIENT
    ES_CLIENT --> ES_DOC
    ES_DOC --> ES

    %% Rejection Log handling flow from Elasticsearch Client
    ES_CLIENT --> RL1
    RL1 --> RL2
    RL1 --> METRIC

    %% Metrics collection inputs
    DCP --> METRIC
    MAPPER --> METRIC
    BATCH --> METRIC
    ES_CLIENT --> METRIC
    ES_DOC --> METRIC

    METRIC --> MON

    %% Configuration used by Elasticsearch Client as well
    CONFIG --> ES_CLIENT

    %% Click Events
    click DCP "https://github.com/trendyol/go-dcp-elasticsearch/blob/master/connector.go"
    click MAPPER "https://github.com/trendyol/go-dcp-elasticsearch/blob/master/mapper.go"
    click CONFIG "https://github.com/trendyol/go-dcp-elasticsearch/blob/master/config/config.go"
    click CE1 "https://github.com/trendyol/go-dcp-elasticsearch/blob/master/couchbase/event.go"
    click CE2 "https://github.com/trendyol/go-dcp-elasticsearch/blob/master/dcp_event_handler.go"
    click ES_CLIENT "https://github.com/trendyol/go-dcp-elasticsearch/blob/master/elasticsearch/client/client.go"
    click BATCH "https://github.com/trendyol/go-dcp-elasticsearch/blob/master/elasticsearch/bulk/bulk.go"
    click ES_DOC "https://github.com/trendyol/go-dcp-elasticsearch/blob/master/elasticsearch/document/document.go"
    click RL1 "https://github.com/trendyol/go-dcp-elasticsearch/blob/master/elasticsearch/rejection_log_sink_response_handler.go"
    click RL2 "https://github.com/trendyol/go-dcp-elasticsearch/blob/master/elasticsearch/sink_response_handler.go"
    click METRIC "https://github.com/trendyol/go-dcp-elasticsearch/blob/master/metric/collector.go"
    click CICD "https://github.com/trendyol/go-dcp-elasticsearch/tree/master/.github/workflows"

    %% Styles
    classDef external fill:#ADD8E6,stroke:#333,stroke-width:2px;
    classDef internal fill:#C0FFC0,stroke:#080,stroke-width:2px;
    classDef cicd fill:#FFFACD,stroke:#B8860B,stroke-width:2px;
```
