# thanhnm3/vht--demo architecture

> Architecture diagram of the GitHub repository thanhnm3/vht--demo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thanhnm3/vht--demo
- Repository: https://github.com/thanhnm3/vht--demo

## Overview

##### Project type and purpose
- This is a real-time data-synchronization pipeline, not a web UI or monolithic app.
- It uses Change Data Capture (CDC) from Aerospike source clusters, streams events into Kafka, then consumes and writes to Aerospike target clusters.
- It includes Java microservices (producer/consumer), a shared “common” library, a test harness, all orchestrated via Docker Compose and monitored by Prometheus/Grafana.

##### File structure
- Top-level “docker/” holds Dockerfiles and compose definitions for spinning up:
- Aerospike clusters (source & target)
- Kafka & ZooKeeper
- Prometheus & Grafana
- MirrorMaker2 for cross-cluster replication
- “my-data-pipeline/” is a multi-module Maven project:
- common/ – shared services & config loaders
- producer-app/ – CDC adapter & event producer to Kafka
- consumer-app/ – Kafka consumer writing to target Aerospike
- test-runner/ – tools for load generation and data verification
- Scripts at root manage Aerospike lifecycle and quick test runs.

##### Key insights from README
- Three logical tiers: Source DB (Aerospike), Transport (Kafka), Target DB (Aerospike).
- Uses CDC, Kafka for real-time, and Prometheus+Grafana for metrics.
- Demo nature; everything designed to run via Docker Compose for quick setup.

1. Aerospike Source Cluster
2. CDC Module / Producer App (Java)
3. Kafka Cluster (with ZooKeeper)
4. Consumer App (Java)
5. Aerospike Target Cluster
6. Prometheus (metrics collector)
7. Grafana (dashboard)
8. MirrorMaker2 (optional cross-cluster link)
9. Test-Runner (load & verification tool)
- CDC Module reads source DB → publishes Change-Events → Kafka topics
- Consumer App subscribes to Kafka → processes events → writes to target DB
- MirrorMaker2 replicates between Kafka clusters if multi-region
- Applications and Aerospike expose metrics → Prometheus scrapes → Grafana visualizes
- Test-Runner injects or reads events to validate end-to-end
c. Architectural patterns
- Event-driven pipeline (producer → broker → consumer)
- Shared library for config and monitoring (common/)
- Docker Compose for environment orchestration
- Microservices segmented by function (CDC producer vs. consumer)
d. Key technologies & frameworks
- Aerospike (NoSQL clusters)
- Kafka & ZooKeeper (messaging)
- Java + Maven (common, producer-app, consumer-app, test-runner)
- Prometheus & Grafana (monitoring)
- Docker & Docker Compose

- Rectangles for microservices (Producer, Consumer, CDC).
- Rounded rectangles or trapezoids for Prometheus/Grafana.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Docker Compose Environment"
        direction TB
        AeroSrc["Aerospike Source Cluster"]:::db
        Zoo["ZooKeeper"]:::infra
        Kafka["Kafka Cluster"]:::infra
        AeroTgt["Aerospike Target Cluster"]:::db
        MM2["MirrorMaker2"]:::infra
        Prom["Prometheus"]:::monitor
        Graf["Grafana"]:::monitor
    end

    subgraph "Application Services"
        direction TB
        Common["Shared Common Library"]:::lib
        Producer["CDC Module / Producer App"]:::service
        Consumer["Consumer App"]:::service
        TR["Test-Runner"]:::service
    end

    %% Data Flow
    AeroSrc -->|CDC Events| Producer
    Producer -->|Kafka JSON event| Kafka
    Kafka -->|subscribe| Consumer
    Consumer -->|writes events| AeroTgt

    %% MirrorMaker2 Replication
    Kafka -->|input topics| MM2
    MM2 -->|replicated topics| Kafka

    %% Test-Runner interactions
    TR -->|load generation| AeroSrc
    TR -->|verification reads| AeroTgt

    %% Monitoring
    AeroSrc -->|"metrics scrape"| Prom
    AeroTgt -->|"metrics scrape"| Prom
    Producer -->|"metrics scrape"| Prom
    Consumer -->|"metrics scrape"| Prom
    Kafka -->|"metrics scrape"| Prom
    Zoo -->|"metrics scrape"| Prom
    MM2 -->|"metrics scrape"| Prom
    Prom -->|"dashboard data"| Graf

    %% Click Events
    click AeroSrc "https://github.com/thanhnm3/vht--demo/blob/main/docker/Dockerfile.aerospike1"
    click AeroTgt "https://github.com/thanhnm3/vht--demo/blob/main/docker/Dockerfile.aerospike2"
    click Zoo "https://github.com/thanhnm3/vht--demo/blob/main/docker/docker-compose-pipeline.yml"
    click Kafka "https://github.com/thanhnm3/vht--demo/blob/main/docker/docker-compose-pipeline.yml"
    click MM2 "https://github.com/thanhnm3/vht--demo/blob/main/docker/mm2/mm2.properties"
    click Prom "https://github.com/thanhnm3/vht--demo/blob/main/docker/etc/prometheus/prometheus.yml"
    click Graf "https://github.com/thanhnm3/vht--demo/tree/main/docker/etc/grafana/dashboards"
    click Common "https://github.com/thanhnm3/vht--demo/tree/main/my-data-pipeline/common"
    click Producer "https://github.com/thanhnm3/vht--demo/tree/main/my-data-pipeline/producer-app"
    click Consumer "https://github.com/thanhnm3/vht--demo/tree/main/my-data-pipeline/consumer-app"
    click TR "https://github.com/thanhnm3/vht--demo/tree/main/my-data-pipeline/test-runner"

    %% Styles
    classDef db fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef infra fill:#FFA500,stroke:#333,stroke-width:1px
    classDef service fill:#90EE90,stroke:#333,stroke-width:1px
    classDef lib fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef monitor fill:#98FB98,stroke:#333,stroke-width:1px
```
