How trendyol/go-dcp-kafka works
Main Components
a) Core Components:
• Couchbase Database (Source)
• DCP Protocol Handler
• Connector Core
• Kafka Producer
• Kafka Cluster (Destination)
• Metadata Storage (Couchbase/Kafka/File)
b) Supporting Components:
• Configuration Management
• Metrics Collection System
• Event Mapping System
• Batch Processing System
┌─ Source Layer ─────┐ ┌─ Processing Layer ──────┐ ┌─ Destination Layer ─┐
• Couchbase │──▶│ DCP Protocol ────┼──▶│ Kafka Cluster │
• Database │ │ Handler │ │ │
• ┘ │ ▼ │ └──────────────────────┘
• Connector Core │
• ▼ │
• Event Mapper │
• ▼ │
• Batch Processor │
• ┘
Key Interactions to Highlight
• DCP event flow (Couchbase → DCP Handler)
• Document transformation flow (DCP Handler → Event Mapper)
• Batch processing flow (Event Mapper → Batch Processor)
• Kafka message production (Batch Processor → Kafka)
Important Subsystems
a) Configuration Management:
• How configuration flows to different components
• Different configuration options (Kafka, DCP, Metadata)
b) Batch Processing:
• Batch size limits
• Batch timing mechanism
• Compression options
c) Event Mapping:
• Demonstrate the transformation pipeline
• Collection-to-topic mapping
Monitoring & Metrics
A separate section showing:
• Prometheus metrics collection
• Grafana dashboard integration
• Key performance indicators
• Latency metrics
• Batch processing metrics
• DCP metrics
Additional Details
Callouts for:
• Security configurations (SSL/TLS, SCRAM)
• Scaling capabilities
• Error handling paths
• Checkpoint management