# creaivelabs-test/kafka-payments-pipeline-benchmark architecture

> Architecture diagram of the GitHub repository creaivelabs-test/kafka-payments-pipeline-benchmark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/creaivelabs-test/kafka-payments-pipeline-benchmark
- Repository: https://github.com/creaivelabs-test/kafka-payments-pipeline-benchmark

## Overview

This repository benchmarks a real-time payments pipeline, measuring throughput and latency across synthetic or IBM TabFormer transaction runs. The principal workflow is Generator → Kafka topics → Enricher → Scorer → Ledger → PostgreSQL; Redis supports enrichment lookups, and the Scorer maintains velocity state. The README documents these runtime stages and optional cluster/fault scenarios; the only sampled source is the IBM evaluation script, so other implementation links are documentation-backed rather than source-verified.

Main entry points: [`src/main/java`](https://github.com/creaivelabs-test/kafka-payments-pipeline-benchmark/tree/main/src/main/java) (Transaction enricher), [`scripts/ibm_eval.py`](https://github.com/creaivelabs-test/kafka-payments-pipeline-benchmark/blob/main/scripts/ibm_eval.py) (IBM ledger evaluator)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_pipeline["Transaction Pipeline"]
  node_generator["Transaction generator"]
  node_kafka["Kafka topics"]
  node_enricher["Transaction enricher"]
  node_scorer["Velocity scorer"]
  node_ledger["Ledger writer"]
end

subgraph group_state["State and Storage"]
  node_redis[("Enrichment cache")]
  node_postgres[("Payments database")]
  node_rocksdb[("Scorer state store")]
end

subgraph group_evaluation["Benchmark Evaluation"]
  node_ibmdata["IBM TabFormer data"]
  node_ibmeval["IBM ledger evaluator<br/>[ibm_eval.py]"]
  node_fraudreport["Fraud metrics<br/>[ibm_eval.py]"]
end

node_operator(("Benchmark operator"))

node_operator -->|"starts run"| node_generator
node_generator -->|"publishes transactions"| node_kafka
node_kafka -->|"supplies transactions"| node_enricher
node_enricher -->|"checks cache"| node_redis
node_enricher -->|"loads cache misses"| node_postgres
node_enricher -->|"publishes enriched"| node_kafka
node_kafka -->|"supplies enriched"| node_scorer
node_scorer -->|"stores velocity state"| node_rocksdb
node_scorer -->|"publishes decisions"| node_kafka
node_kafka -->|"supplies decisions"| node_ledger
node_ledger -->|"writes ledger rows"| node_postgres
node_operator -->|"selects replay data"| node_ibmdata
node_ibmdata -->|"provides source rows"| node_ibmeval
node_ibmeval -->|"reads ledger"| node_postgres
node_ibmeval -->|"writes metrics"| node_fraudreport

click node_generator "https://github.com/creaivelabs-test/kafka-payments-pipeline-benchmark/tree/main/src/main/java"
click node_enricher "https://github.com/creaivelabs-test/kafka-payments-pipeline-benchmark/tree/main/src/main/java"
click node_scorer "https://github.com/creaivelabs-test/kafka-payments-pipeline-benchmark/tree/main/src/main/java"
click node_ledger "https://github.com/creaivelabs-test/kafka-payments-pipeline-benchmark/tree/main/src/main/java"
click node_ibmeval "https://github.com/creaivelabs-test/kafka-payments-pipeline-benchmark/blob/main/scripts/ibm_eval.py"
click node_fraudreport "https://github.com/creaivelabs-test/kafka-payments-pipeline-benchmark/blob/main/scripts/ibm_eval.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_generator,node_kafka,node_enricher,node_scorer,node_ledger toneBlue
class node_redis,node_postgres,node_rocksdb toneAmber
class node_ibmdata,node_ibmeval,node_fraudreport toneMint
class node_operator toneIndigo
```

## Components

### Transaction Pipeline

- [Transaction generator](https://github.com/creaivelabs-test/kafka-payments-pipeline-benchmark/tree/main/src/main/java): `src/main/java`
- **Kafka topics**
- [Transaction enricher](https://github.com/creaivelabs-test/kafka-payments-pipeline-benchmark/tree/main/src/main/java): `src/main/java`
- [Velocity scorer](https://github.com/creaivelabs-test/kafka-payments-pipeline-benchmark/tree/main/src/main/java): `src/main/java`
- [Ledger writer](https://github.com/creaivelabs-test/kafka-payments-pipeline-benchmark/tree/main/src/main/java): `src/main/java`

### State and Storage

- **Enrichment cache**
- **Payments database**
- **Scorer state store**

### Benchmark Evaluation

- **IBM TabFormer data**
- [IBM ledger evaluator](https://github.com/creaivelabs-test/kafka-payments-pipeline-benchmark/blob/main/scripts/ibm_eval.py): `scripts/ibm_eval.py`
- [Fraud metrics](https://github.com/creaivelabs-test/kafka-payments-pipeline-benchmark/blob/main/scripts/ibm_eval.py): `scripts/ibm_eval.py`

### Other components

- **Benchmark operator**

## Connections

- Benchmark operator → Transaction generator: starts run (evidence: `README.md`)
- Transaction generator → Kafka topics: publishes transactions (evidence: `README.md`)
- Kafka topics → Transaction enricher: supplies transactions (evidence: `README.md`)
- Transaction enricher → Enrichment cache: checks cache (evidence: `README.md`)
- Transaction enricher → Payments database: loads cache misses (evidence: `README.md`)
- Transaction enricher → Kafka topics: publishes enriched (evidence: `README.md`)
- Kafka topics → Velocity scorer: supplies enriched (evidence: `README.md`)
- Velocity scorer → Scorer state store: stores velocity state (evidence: `README.md`)
- Velocity scorer → Kafka topics: publishes decisions (evidence: `README.md`)
- Kafka topics → Ledger writer: supplies decisions (evidence: `README.md`)
- Ledger writer → Payments database: writes ledger rows (evidence: `README.md`)
- Benchmark operator → IBM TabFormer data: selects replay data (evidence: `README.md`)
- IBM TabFormer data → IBM ledger evaluator: provides source rows (evidence: `scripts/ibm_eval.py`)
- IBM ledger evaluator → Payments database: reads ledger (evidence: `scripts/ibm_eval.py`)
- IBM ledger evaluator → Fraud metrics: writes metrics (evidence: `scripts/ibm_eval.py`)