How girishn/real-time-loyalty-platform works
What this project is (from file tree + README)
This repo is primarily infrastructure-as-code for a real-time, event-driven “Loyalty Platform” on AWS.
Evidence:
• infra/terraform/** with modules:
• msk_cluster/ → AWS MSK (Kafka)
• ec2_client/ → an EC2-based client (likely for producing/consuming events, testing, or admin tooling)
• root main.tf, provider.tf, variables.tf, outputs.tf → orchestrates the full stack
• docs/kafka-topics.md → domain/event design is important (topics are first-class architecture)
• README describes runtime components: MSK, Flink on EKS, DynamoDB, API Gateway + Lambda, S3 (+ Athena).
So: this is a streaming/event-sourcing-ish architecture (Kafka backbone), with stateful stream processing (Flink), a serving store (DynamoDB), and a query API (API Gateway/Lambda), plus a data lake export path (S3/Athena).
Start from Terraform as the source of truth
Because there’s no application code here, Terraform defines the real deployed boundaries. Parse:
• infra/terraform/main.tf → what modules/resources get created and how they connect
• infra/terraform/msk_cluster/* → cluster topology, networking, security groups, auth (IAM/SASL/SCRAM), broker subnets, etc.
• infra/terraform/ec2_client/* → what this instance can reach (MSK? EKS? public internet?), what it’s for
• variables.tf/outputs.tf → reveals key endpoints and integration points (bootstrap brokers, VPC IDs, security groups, cluster ARN, etc.)
Use docs/kafka-topics.md to define the “contract layer”
• Topic names (or grouped by domain)
• Producers and consumers per topic
• Key semantics: partitions, keys (customerId?), retention, compaction (if relevant)
This often matters more than “which microservice talks to which” in streaming systems.
Event Producers
• “Retail/POS”, “E-commerce”, “Payment processor”, “Partner systems”
• (If the EC2 client is a producer/consumer tool, show it here explicitly: “EC2 Kafka Client (test/ops)”)
AWS MSK (Kafka backbone)
• Brokers + ZooKeeper/KRaft (depending on config; MSK may manage ZooKeeper or KRaft—reflect what Terraform provisions)
• Topics (group by domain): e.g., transactions, accrual, redemption, fraud-signals, points-ledger-events, etc. (from docs/kafka-topics.md)
Flink on EKS
• EKS cluster
• Flink components: JobManager, TaskManagers
• Stateful processing responsibilities:
• Points calculation
• Fraud detection
DynamoDB (Serving / current state)
• This is the low-latency source for API reads.
API Layer
• API Gateway
• Lambda: “GetPointsBalance”, “GetTransactions”, etc.
• AuthN/AuthZ (Cognito/IAM authorizer) if applicable (only add if known; otherwise show “Auth (TBD)” as a boundary)
Historical / Analytics path
• S3 bucket (event export)
• Athena (query)
• (Optionally Glue Catalog if used; not shown in repo—only add if real)
Networking boundary
• VPC
• Public/private subnets
• NAT/IGW (if present in Terraform)
• Security groups (especially: who can talk to MSK brokers, EKS, DynamoDB endpoints)
IAM / Security
• IAM roles for:
• EKS worker nodes / IRSA service accounts (Flink)
• Lambda execution
• EC2 client
• MSK auth mechanism (IAM/SASL) if used
Observability (if present or intended)
• CloudWatch logs/metrics
• Prometheus/Grafana (often with EKS)
Real-time event flow
• Flink → produces derived events back to MSK (optional but common): e.g., points-updated, fraud-alerts
Query flow (CQRS-ish separation)
• Client apps → API Gateway → Lambda → DynamoDB
• If Lambda also emits events (e.g., redemption request), show Lambda → MSK as well (only if true)
Batch/analytics flow
• MSK → S3 (via connector/export job; name it explicitly: “Kafka Connect S3 Sink” or “Flink export job”)
• Athena → S3 (query)
Architectural patterns to call out
Make these visible with labels (not paragraphs):
• Event-driven architecture: MSK as the integration spine
• Stream processing / Stateful computation: Flink jobs
• Materialized view / Serving store: DynamoDB holds “current points balance”
• CQRS (likely): write-side via events + processing; read-side via API querying DynamoDB
• Append-only ledger vs. current balance (if topic/table design implies it): show both if it exists (ledger events + balance table)
System Context (Level 0)
• Only: Producers, consumers, analysts
Container / Runtime Architecture (Level 1)
• MSK, EKS/Flink, DynamoDB, API GW/Lambda, S3/Athena, Export component
Deployment / AWS Infrastructure (Level 2)
• *where it runs*:
• VPC, subnets, security groups
• MSK cluster placement
• EKS node groups
• Whether Lambda is VPC-attached (important for DynamoDB/MSK access patterns)
This is where Terraform maps most directly.
Since the repo is infra/terraform, annotate each major component with its module source:
• “MSK Cluster (infra/terraform/msk_cluster)”
• “EC2 Client (infra/terraform/ec2_client)”
• “Root stack (infra/terraform/main.tf)”
Labels
• “AWS MSK (Kafka) — event backbone”
• “Flink on EKS — points calc + fraud detection”
• “DynamoDB — points serving store”
• “API Gateway + Lambda — customer query API”
• “S3 — event archive; Athena — analytics”
• “Produce”, “Consume”, “Enrich”, “Upsert”, “Query”, “Export”
What to add to make it “principal-engineer accurate”
• Auth to MSK (IAM/SASL/SCRAM) and which principals use it (Flink role, EC2 role)
• Network reachability: private brokers, VPC-only access, peering, etc.
• Scaling units:
• MSK: number of brokers, partitions (conceptually)
• Flink: parallelism / task slots (conceptually)
• Failure handling:
• DLQ topic(s) or retry topics (if defined in kafka-topics.md)
• Exactly-once / at-least-once semantics (label if known)
If not known from repo/docs, include them as “(TBD)” rather than inventing details.
EKS cluster container with Flink JobManager/TaskManagers.
• Producers → MSK
• MSK → Flink
• Flink → DynamoDB
• (Optional) Flink → MSK derived topics
MSK → Export component → S3 → Athena analytics path.
7. EC2 client (from Terraform module) and connect it to what it actually accesses (likely MSK).