How akhiljagtap/payments-pipeline works
This repository implements a payments-event streaming pipeline: it generates valid and deliberately faulty events, publishes them to Kafka, ingests them into a Bronze Delta table, then validates and separates accepted events into Silver and rejected events into quarantine. Gold aggregation jobs, query scripts, and data-quality checks provide downstream summaries and inspection; Airflow can trigger the daily summary. The sampled sources establish the producer-to-Bronze-to-Silver flow and the summary trigger. Gold job internals and most query/check behavior were not sampled, so their connections are not inferred.
Main entry points: kafka_producer.py Kafka producer · silver_job.py Silver processing · bronze_job.py Bronze ingestion · event_factory.py Event factory
Components 28
Event generation
- Kafka producer
src/producer/kafka_producer.py - Event factory
src/producer/event_factory.py - Fault ground truth
src/producer/ground_truth.py
Stream ingestion
- Transactions topic
- Bronze ingestion
src/streaming/bronze_job.py - Stream event schema
src/streaming/schemas.py - Bronze Delta table
- Bronze checks
src/streaming/bronze_checks.py - Bronze reader
src/streaming/read_bronze.py
Validation processing
- Silver processing
src/streaming/silver_job.py - Validation rules
src/streaming/validation.py - Silver Delta table
- Quarantine Delta table
- Silver reader
src/streaming/read_silver.py - Card checks
src/streaming/silver_card_check.py - Recent-event checks
src/streaming/silver_recent_check.py - Card timeline check
src/streaming/card_timeline_check.py
Analytics and access
- Daily summary job
src/streaming/gold_daily_summary.py - Fraud velocity job
src/streaming/gold_fraud_velocity.py - Merchant metrics job
src/streaming/gold_merchant_metrics.py - Daily summary reader
src/streaming/read_gold_daily_summary.py - Fraud velocity reader
src/streaming/read_gold_fraud_velocity.py - Metrics reader
src/streaming/read_gold_metrics.py - Airflow DAG
airflow/dags/daily_summary_dag.py
Shared components
- Payment event model
src/common/schemas.py - Runtime settings
src/common/config.py - Logging
src/common/logger.py
Other components
- Pipeline operator
Connections 16
- Kafka producer → Event factory: creates events
src/producer/kafka_producer.py - Event factory → Payment event model: builds events
src/producer/event_factory.py - Kafka producer → Fault ground truth: records faults
src/producer/kafka_producer.py - Kafka producer → Runtime settings: reads settings
src/producer/kafka_producer.py - Kafka producer → Logging: gets logger
src/producer/kafka_producer.py - Fault ground truth → Runtime settings: imports settings
src/producer/ground_truth.py - Kafka producer → Transactions topic: publishes events
src/producer/kafka_producer.py - Bronze ingestion → Transactions topic: reads stream
src/streaming/bronze_job.py - Bronze ingestion → Stream event schema: parses with
src/streaming/bronze_job.py - Bronze ingestion → Bronze Delta table: writes rows
src/streaming/bronze_job.py - Silver processing → Bronze Delta table: reads stream
src/streaming/silver_job.py - Silver processing → Validation rules: applies rules
src/streaming/silver_job.py - Silver processing → Silver Delta table: upserts accepted
src/streaming/silver_job.py - Silver processing → Quarantine Delta table: writes rejected
src/streaming/silver_job.py - Pipeline operator → Airflow DAG: triggers DAG
- Airflow DAG → Daily summary job: runs job
airflow/dags/daily_summary_dag.py