How thanhnm3/vht--demo works
Project type and purpose
• This is a real-time data-synchronization pipeline, not a web UI or monolithic app.
• It uses Change Data Capture (CDC) from Aerospike source clusters, streams events into Kafka, then consumes and writes to Aerospike target clusters.
• It includes Java microservices (producer/consumer), a shared “common” library, a test harness, all orchestrated via Docker Compose and monitored by Prometheus/Grafana.
File structure
• Top-level “docker/” holds Dockerfiles and compose definitions for spinning up:
• Aerospike clusters (source & target)
• Kafka & ZooKeeper
• Prometheus & Grafana
• MirrorMaker2 for cross-cluster replication
• “my-data-pipeline/” is a multi-module Maven project:
• common/ – shared services & config loaders
• producer-app/ – CDC adapter & event producer to Kafka
• consumer-app/ – Kafka consumer writing to target Aerospike
• test-runner/ – tools for load generation and data verification
• Scripts at root manage Aerospike lifecycle and quick test runs.
Key insights from README
• Three logical tiers: Source DB (Aerospike), Transport (Kafka), Target DB (Aerospike).
• Uses CDC, Kafka for real-time, and Prometheus+Grafana for metrics.
• Demo nature; everything designed to run via Docker Compose for quick setup.
1. Aerospike Source Cluster
2. CDC Module / Producer App (Java)
3. Kafka Cluster (with ZooKeeper)
4. Consumer App (Java)
5. Aerospike Target Cluster
6. Prometheus (metrics collector)
7. Grafana (dashboard)
8. MirrorMaker2 (optional cross-cluster link)
9. Test-Runner (load & verification tool)
• CDC Module reads source DB → publishes Change-Events → Kafka topics
• Consumer App subscribes to Kafka → processes events → writes to target DB
• MirrorMaker2 replicates between Kafka clusters if multi-region
• Applications and Aerospike expose metrics → Prometheus scrapes → Grafana visualizes
• Test-Runner injects or reads events to validate end-to-end
c. Architectural patterns
• Event-driven pipeline (producer → broker → consumer)
• Shared library for config and monitoring (common/)
• Docker Compose for environment orchestration
• Microservices segmented by function (CDC producer vs. consumer)
d. Key technologies & frameworks
• Aerospike (NoSQL clusters)
• Kafka & ZooKeeper (messaging)
• Java + Maven (common, producer-app, consumer-app, test-runner)
• Prometheus & Grafana (monitoring)
• Docker & Docker Compose
• Rectangles for microservices (Producer, Consumer, CDC).
• Rounded rectangles or trapezoids for Prometheus/Grafana.