# akhiljagtap/payments-pipeline architecture

> Architecture diagram of the GitHub repository akhiljagtap/payments-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/akhiljagtap/payments-pipeline
- Repository: https://github.com/akhiljagtap/payments-pipeline

## Overview

This repository implements a payments-event streaming pipeline: it generates valid and deliberately faulty events, publishes them to Kafka, ingests them into a Bronze Delta table, then validates and separates accepted events into Silver and rejected events into quarantine. Gold aggregation jobs, query scripts, and data-quality checks provide downstream summaries and inspection; Airflow can trigger the daily summary. The sampled sources establish the producer-to-Bronze-to-Silver flow and the summary trigger. Gold job internals and most query/check behavior were not sampled, so their connections are not inferred.

Main entry points: [`src/producer/kafka_producer.py`](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/producer/kafka_producer.py) (Kafka producer), [`src/streaming/silver_job.py`](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/silver_job.py) (Silver processing), [`src/streaming/bronze_job.py`](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/bronze_job.py) (Bronze ingestion), [`src/producer/event_factory.py`](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/producer/event_factory.py) (Event factory)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_generation["Event generation"]
  node_producer["Kafka producer<br/>[kafka_producer.py]"]
  node_factory["Event factory<br/>[event_factory.py]"]
  node_ground_truth["Fault ground truth<br/>[ground_truth.py]"]
end

subgraph group_ingestion["Stream ingestion"]
  node_kafka["Transactions topic"]
  node_bronze_job["Bronze ingestion<br/>[bronze_job.py]"]
  node_stream_schema["Stream event schema<br/>[schemas.py]"]
  node_bronze[("Bronze Delta table")]
  node_bronze_checks["Bronze checks<br/>[bronze_checks.py]"]
  node_bronze_reader["Bronze reader<br/>[read_bronze.py]"]
end

subgraph group_processing["Validation processing"]
  node_silver_job["Silver processing<br/>[silver_job.py]"]
  node_validation["Validation rules<br/>[validation.py]"]
  node_silver[("Silver Delta table")]
  node_quarantine[("Quarantine Delta table")]
  node_silver_reader["Silver reader<br/>[read_silver.py]"]
  node_card_check["Card checks"]
  node_recent_check["Recent-event checks"]
  node_timeline_check["Card timeline check"]
end

subgraph group_analytics["Analytics and access"]
  node_daily_gold["Daily summary job"]
  node_fraud_gold["Fraud velocity job"]
  node_merchant_gold["Merchant metrics job"]
  node_daily_reader["Daily summary reader"]
  node_fraud_reader["Fraud velocity reader"]
  node_metrics_reader["Metrics reader"]
  node_airflow["Airflow DAG"]
end

subgraph group_shared["Shared components"]
  node_event_model["Payment event model<br/>[schemas.py]"]
  node_settings["Runtime settings<br/>[config.py]"]
  node_logger["Logging<br/>[logger.py]"]
end

node_operator(("Pipeline operator"))

node_producer -->|"creates events"| node_factory
node_factory -->|"builds events"| node_event_model
node_producer -->|"records faults"| node_ground_truth
node_producer -->|"reads settings"| node_settings
node_producer -->|"gets logger"| node_logger
node_ground_truth -->|"imports settings"| node_settings
node_producer -->|"publishes events"| node_kafka
node_bronze_job -->|"reads stream"| node_kafka
node_bronze_job -->|"parses with"| node_stream_schema
node_bronze_job -->|"writes rows"| node_bronze
node_silver_job -->|"reads stream"| node_bronze
node_silver_job -->|"applies rules"| node_validation
node_silver_job -->|"upserts accepted"| node_silver
node_silver_job -->|"writes rejected"| node_quarantine
node_operator -.->|"triggers DAG"| node_airflow
node_airflow -->|"runs job"| node_daily_gold

click node_producer "https://github.com/akhiljagtap/payments-pipeline/blob/main/src/producer/kafka_producer.py"
click node_factory "https://github.com/akhiljagtap/payments-pipeline/blob/main/src/producer/event_factory.py"
click node_event_model "https://github.com/akhiljagtap/payments-pipeline/blob/main/src/common/schemas.py"
click node_ground_truth "https://github.com/akhiljagtap/payments-pipeline/blob/main/src/producer/ground_truth.py"
click node_settings "https://github.com/akhiljagtap/payments-pipeline/blob/main/src/common/config.py"
click node_logger "https://github.com/akhiljagtap/payments-pipeline/blob/main/src/common/logger.py"
click node_bronze_job "https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/bronze_job.py"
click node_stream_schema "https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/schemas.py"
click node_bronze_checks "https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/bronze_checks.py"
click node_bronze_reader "https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/read_bronze.py"
click node_silver_job "https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/silver_job.py"
click node_validation "https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/validation.py"
click node_silver_reader "https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/read_silver.py"
click node_card_check "https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/silver_card_check.py"
click node_recent_check "https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/silver_recent_check.py"
click node_timeline_check "https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/card_timeline_check.py"
click node_daily_gold "https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/gold_daily_summary.py"
click node_fraud_gold "https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/gold_fraud_velocity.py"
click node_merchant_gold "https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/gold_merchant_metrics.py"
click node_daily_reader "https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/read_gold_daily_summary.py"
click node_fraud_reader "https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/read_gold_fraud_velocity.py"
click node_metrics_reader "https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/read_gold_metrics.py"
click node_airflow "https://github.com/akhiljagtap/payments-pipeline/blob/main/airflow/dags/daily_summary_dag.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_producer,node_factory,node_ground_truth toneBlue
class node_kafka,node_bronze_job,node_stream_schema,node_bronze,node_bronze_checks,node_bronze_reader toneAmber
class node_silver_job,node_validation,node_silver,node_quarantine,node_silver_reader,node_card_check,node_recent_check,node_timeline_check toneMint
class node_daily_gold,node_fraud_gold,node_merchant_gold,node_daily_reader,node_fraud_reader,node_metrics_reader,node_airflow toneRose
class node_event_model,node_settings,node_logger,node_operator toneIndigo
```

## Components

### Event generation

- [Kafka producer](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/producer/kafka_producer.py): `src/producer/kafka_producer.py`
- [Event factory](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/producer/event_factory.py): `src/producer/event_factory.py`
- [Fault ground truth](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/producer/ground_truth.py): `src/producer/ground_truth.py`

### Stream ingestion

- **Transactions topic**
- [Bronze ingestion](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/bronze_job.py): `src/streaming/bronze_job.py`
- [Stream event schema](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/schemas.py): `src/streaming/schemas.py`
- **Bronze Delta table**
- [Bronze checks](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/bronze_checks.py): `src/streaming/bronze_checks.py`
- [Bronze reader](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/read_bronze.py): `src/streaming/read_bronze.py`

### Validation processing

- [Silver processing](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/silver_job.py): `src/streaming/silver_job.py`
- [Validation rules](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/validation.py): `src/streaming/validation.py`
- **Silver Delta table**
- **Quarantine Delta table**
- [Silver reader](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/read_silver.py): `src/streaming/read_silver.py`
- [Card checks](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/silver_card_check.py): `src/streaming/silver_card_check.py`
- [Recent-event checks](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/silver_recent_check.py): `src/streaming/silver_recent_check.py`
- [Card timeline check](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/card_timeline_check.py): `src/streaming/card_timeline_check.py`

### Analytics and access

- [Daily summary job](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/gold_daily_summary.py): `src/streaming/gold_daily_summary.py`
- [Fraud velocity job](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/gold_fraud_velocity.py): `src/streaming/gold_fraud_velocity.py`
- [Merchant metrics job](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/gold_merchant_metrics.py): `src/streaming/gold_merchant_metrics.py`
- [Daily summary reader](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/read_gold_daily_summary.py): `src/streaming/read_gold_daily_summary.py`
- [Fraud velocity reader](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/read_gold_fraud_velocity.py): `src/streaming/read_gold_fraud_velocity.py`
- [Metrics reader](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/streaming/read_gold_metrics.py): `src/streaming/read_gold_metrics.py`
- [Airflow DAG](https://github.com/akhiljagtap/payments-pipeline/blob/main/airflow/dags/daily_summary_dag.py): `airflow/dags/daily_summary_dag.py`

### Shared components

- [Payment event model](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/common/schemas.py): `src/common/schemas.py`
- [Runtime settings](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/common/config.py): `src/common/config.py`
- [Logging](https://github.com/akhiljagtap/payments-pipeline/blob/main/src/common/logger.py): `src/common/logger.py`

### Other components

- **Pipeline operator**

## Connections

- Kafka producer → Event factory: creates events (evidence: `src/producer/kafka_producer.py`)
- Event factory → Payment event model: builds events (evidence: `src/producer/event_factory.py`)
- Kafka producer → Fault ground truth: records faults (evidence: `src/producer/kafka_producer.py`)
- Kafka producer → Runtime settings: reads settings (evidence: `src/producer/kafka_producer.py`)
- Kafka producer → Logging: gets logger (evidence: `src/producer/kafka_producer.py`)
- Fault ground truth → Runtime settings: imports settings (evidence: `src/producer/ground_truth.py`)
- Kafka producer → Transactions topic: publishes events (evidence: `src/producer/kafka_producer.py`)
- Bronze ingestion → Transactions topic: reads stream (evidence: `src/streaming/bronze_job.py`)
- Bronze ingestion → Stream event schema: parses with (evidence: `src/streaming/bronze_job.py`)
- Bronze ingestion → Bronze Delta table: writes rows (evidence: `src/streaming/bronze_job.py`)
- Silver processing → Bronze Delta table: reads stream (evidence: `src/streaming/silver_job.py`)
- Silver processing → Validation rules: applies rules (evidence: `src/streaming/silver_job.py`)
- Silver processing → Silver Delta table: upserts accepted (evidence: `src/streaming/silver_job.py`)
- Silver processing → Quarantine Delta table: writes rejected (evidence: `src/streaming/silver_job.py`)
- Pipeline operator → Airflow DAG: triggers DAG
- Airflow DAG → Daily summary job: runs job (evidence: `airflow/dags/daily_summary_dag.py`)
