How trendyol/go-dcp-elasticsearch works
Project Type and Purpose
• This is a Go-based connector designed to integrate Couchbase (as the source of change events) with Elasticsearch (the destination for indexing documents).
• The README and file structure indicate high performance and customizable mapping as key features. It supports batching, custom routing, and synchronization between the two systems.
File Structure
• Top-Level Components:
• Configuration: The “config” directory contains go files for configuration settings.
• Connector Logic: Files like connector.go and mapper.go indicate central processing logic to manage DCP events and map them into Elasticsearch actions.
• Elasticsearch-specific modules: Organized under the “elasticsearch” directory, including subdirectories for “client”, “document”, and “bulk” processing.
• Couchbase-related files: Located under the “couchbase” directory.
• Examples: The “example” directory contains various subprojects (e.g., default-mapper, doc-update, grafana) that illustrate different usage scenarios.
• Testing and Benchmarking: The “test” and “benchmark” directories highlight integration tests and performance benchmarking.
• CI/CD Integration: The .github directory with workflows shows continuous integration, testing, and release pipelines.
README for Additional Insights
• The README describes core functionality such as processing DCP events (e.g., mutation, expiration, deletion) to perform index or delete operations in Elasticsearch.
• It points out performance advantages, detailed configuration options (batch size, compression, etc.), and exposes metrics for observability.
• References to external tools like Grafana and Prometheus for monitoring suggest the importance of observability within the system.
a. Main Components:
• External Systems:
• Couchbase – Source of DCP events.
• Elasticsearch – Destination for document indexing.
• Monitoring Systems – Prometheus and Grafana for collecting metrics and visualizing performance.
• Internal Components:
• DCP Connector: Listens to and streams events from Couchbase.
• Mapper Module: Transforms Couchbase events into Elasticsearch actions/documents.
• Elasticsearch Client: Handles bulk requests, manages connection pooling, and processes indexing operations.
• Configuration Manager: Reads and applies configurations (from config files, environment variables, etc.).
• Metrics & Logging Modules: Capture metrics like request latency and action success/error counts; also includes handling of rejection logs.
• Batch Processor: Manages batching of events (by message count, size, and durations) before sending to Elasticsearch.
• CI/CD Infrastructure: Represented by GitHub workflows and Dockerfiles, illustrating how the system is built, tested, and deployed.
b. Relationships and Interactions:
• Data Flow:
1. Couchbase emits DCP events.
2. The DCP Connector receives these events and routes them to the Mapper.
3. The Mapper converts events to Elasticsearch action documents.
4. The Batch Processor aggregates these actions based on configured limits.
5. The Elasticsearch Client sends batched documents to Elasticsearch.
6. Metrics and logs are collected throughout the process and pushed to Prometheus/Grafana.
• Control Flow:
• Configuration settings (e.g., batch sizes, connection parameters) are applied in the Connector and Elasticsearch Client.
• Error handling (such as dealing with rejections) is processed via dedicated handlers.
c. Architectural Patterns and Design Principles:
• Modular Design: Each major function (data ingestion, mapping, batching, indexing) is handled by distinct modules.
• Real-Time Streaming: Designed for near real‑time processing of data changes.
• Batch Processing: Optimization of throughput and resource usage by aggregating events.
• Extensibility: Custom mapper functions (shown in various examples) allow for tailored transformations.
• Resilience and Observability: Metrics collection and logging (including rejection logs) ensure the system is robust and manageable.
d. Relevant Technologies and Frameworks:
• Go Programming Language with its standard library and concurrency model.
• Docker for containerization (as seen in multiple Dockerfiles and docker-compose.yml).
• GitHub Actions for CI/CD pipelines.
• Elasticsearch – utilized via custom clients and bulk APIs.
• Prometheus and Grafana – for monitoring and visualization of performance metrics.
• Use clear, labeled components for:
• Couchbase (DCP source)
• DCP Connector Module
• Mapper Module for event transformation
• Batch Processor managing request accumulation
• Elasticsearch Client handling document indexing
• Configuration Manager handling various configuration parameters
• Monitoring Components for metrics collection
• Data flow from Couchbase → Connector → Mapper → Batch Processor → Elasticsearch.
• Metrics/logs flowing to Monitoring tools.