# girishn/real-time-loyalty-platform architecture

> Architecture diagram of the GitHub repository girishn/real-time-loyalty-platform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/girishn/real-time-loyalty-platform
- Repository: https://github.com/girishn/real-time-loyalty-platform

## Overview

##### What this project is (from file tree + README)
This repo is primarily **infrastructure-as-code for a real-time, event-driven “Loyalty Platform” on AWS**.

Evidence:
- `infra/terraform/**` with modules:
- `msk_cluster/` → AWS MSK (Kafka)
- `ec2_client/` → an EC2-based client (likely for producing/consuming events, testing, or admin tooling)
- root `main.tf`, `provider.tf`, `variables.tf`, `outputs.tf` → orchestrates the full stack
- `docs/kafka-topics.md` → domain/event design is important (topics are first-class architecture)
- README describes runtime components: **MSK, Flink on EKS, DynamoDB, API Gateway + Lambda, S3 (+ Athena)**.

So: this is a **streaming/event-sourcing-ish architecture** (Kafka backbone), with **stateful stream processing** (Flink), a **serving store** (DynamoDB), and a **query API** (API Gateway/Lambda), plus a **data lake export path** (S3/Athena).

##### Start from Terraform as the source of truth
Because there’s no application code here, **Terraform defines the real deployed boundaries**. Parse:
- `infra/terraform/main.tf` → what modules/resources get created and how they connect
- `infra/terraform/msk_cluster/*` → cluster topology, networking, security groups, auth (IAM/SASL/SCRAM), broker subnets, etc.
- `infra/terraform/ec2_client/*` → what this instance can reach (MSK? EKS? public internet?), what it’s for
- `variables.tf`/`outputs.tf` → reveals key endpoints and integration points (bootstrap brokers, VPC IDs, security groups, cluster ARN, etc.)

##### Use `docs/kafka-topics.md` to define the “contract layer”
- Topic names (or grouped by domain)
- Producers and consumers per topic
- Key semantics: partitions, keys (customerId?), retention, compaction (if relevant)
This often matters more than “which microservice talks to which” in streaming systems.

##### Event Producers
- “Retail/POS”, “E-commerce”, “Payment processor”, “Partner systems”
- (If the EC2 client is a producer/consumer tool, show it here explicitly: “EC2 Kafka Client (test/ops)”)

##### AWS MSK (Kafka backbone)
- Brokers + ZooKeeper/KRaft (depending on config; MSK may manage ZooKeeper or KRaft—reflect what Terraform provisions)
- Topics (group by domain): e.g., `transactions`, `accrual`, `redemption`, `fraud-signals`, `points-ledger-events`, etc. (from `docs/kafka-topics.md`)

##### Flink on EKS
- EKS cluster
- Flink components: JobManager, TaskManagers
- Stateful processing responsibilities:
- Points calculation
- Fraud detection

##### DynamoDB (Serving / current state)
- This is the low-latency source for API reads.

##### API Layer
- API Gateway
- Lambda: “GetPointsBalance”, “GetTransactions”, etc.
- AuthN/AuthZ (Cognito/IAM authorizer) if applicable (only add if known; otherwise show “Auth (TBD)” as a boundary)

##### Historical / Analytics path
- S3 bucket (event export)
- Athena (query)
- (Optionally Glue Catalog if used; not shown in repo—only add if real)

##### Networking boundary
- VPC
- Public/private subnets
- NAT/IGW (if present in Terraform)
- Security groups (especially: who can talk to MSK brokers, EKS, DynamoDB endpoints)

##### IAM / Security
- IAM roles for:
- EKS worker nodes / IRSA service accounts (Flink)
- Lambda execution
- EC2 client
- MSK auth mechanism (IAM/SASL) if used

##### Observability (if present or intended)
- CloudWatch logs/metrics
- Prometheus/Grafana (often with EKS)

##### Real-time event flow
- Flink → produces derived events back to MSK (optional but common): e.g., `points-updated`, `fraud-alerts`

##### Query flow (CQRS-ish separation)
- Client apps → API Gateway → Lambda → DynamoDB
- If Lambda also emits events (e.g., redemption request), show Lambda → MSK as well (only if true)

##### Batch/analytics flow
- MSK → S3 (via connector/export job; name it explicitly: “Kafka Connect S3 Sink” or “Flink export job”)
- Athena → S3 (query)

##### Architectural patterns to call out
Make these visible with labels (not paragraphs):
- **Event-driven architecture**: MSK as the integration spine
- **Stream processing / Stateful computation**: Flink jobs
- **Materialized view / Serving store**: DynamoDB holds “current points balance”
- **CQRS** (likely): write-side via events + processing; read-side via API querying DynamoDB
- **Append-only ledger vs. current balance** (if topic/table design implies it): show both if it exists (ledger events + balance table)

##### System Context (Level 0)
- Only: Producers, consumers, analysts

##### Container / Runtime Architecture (Level 1)
- MSK, EKS/Flink, DynamoDB, API GW/Lambda, S3/Athena, Export component

##### Deployment / AWS Infrastructure (Level 2)
- *where it runs*:
- VPC, subnets, security groups
- MSK cluster placement
- EKS node groups
- Whether Lambda is VPC-attached (important for DynamoDB/MSK access patterns)
This is where Terraform maps most directly.

Since the repo is `infra/terraform`, annotate each major component with its module source:
- “MSK Cluster (infra/terraform/msk_cluster)”
- “EC2 Client (infra/terraform/ec2_client)”
- “Root stack (infra/terraform/main.tf)”

##### Labels
- “AWS MSK (Kafka) — event backbone”
- “Flink on EKS — points calc + fraud detection”
- “DynamoDB — points serving store”
- “API Gateway + Lambda — customer query API”
- “S3 — event archive; Athena — analytics”

- “Produce”, “Consume”, “Enrich”, “Upsert”, “Query”, “Export”

##### What to add to make it “principal-engineer accurate”
- **Auth** to MSK (IAM/SASL/SCRAM) and which principals use it (Flink role, EC2 role)
- **Network reachability**: private brokers, VPC-only access, peering, etc.
- **Scaling units**:
- MSK: number of brokers, partitions (conceptually)
- Flink: parallelism / task slots (conceptually)
- **Failure handling**:
- DLQ topic(s) or retry topics (if defined in `kafka-topics.md`)
- Exactly-once / at-least-once semantics (label if known)

If not known from repo/docs, include them as “(TBD)” rather than inventing details.

##### EKS cluster container with Flink JobManager/TaskManagers.
- Producers → MSK
- MSK → Flink
- Flink → DynamoDB
- (Optional) Flink → MSK derived topics
##### MSK → Export component → S3 → Athena analytics path.
7. **EC2 client** (from Terraform module) and connect it to what it actually accesses (likely MSK).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% =========================
    %% Level 0: System Context
    %% =========================
    subgraph "System Context (Level 0)"
        direction TB

        POS["Retail / POS Systems"]:::external
        ECOM["E-commerce Platform"]:::external
        PAY["Payment Processor"]:::external
        PARTNER["Partner Systems"]:::external

        CUST["Customer / Mobile App"]:::external
        ANALYST["Data Analyst"]:::external
    end

    PLATFORM["Real-Time Loyalty Platform (AWS)"]:::boundary

    POS -->|"events"| PLATFORM
    ECOM -->|"events"| PLATFORM
    PAY -->|"events"| PLATFORM
    PARTNER -->|"events"| PLATFORM
    CUST -->|"queries"| PLATFORM
    ANALYST -->|"analytics_queries"| PLATFORM

    %% =====================================
    %% Level 1: Runtime / Container Diagram
    %% =====================================
    subgraph "Runtime Architecture (Level 1)"
        direction TB

        %% --- Contract Layer (Topics) ---
        TOPICS["Kafka Topics (Contract Layer)\nexamples: transactions, accrual, redemption,\nfraud-signals, points-ledger-events, points-updated"]:::contract

        %% --- Streaming backbone ---
        subgraph "AWS MSK (Kafka) — event backbone"
            direction TB
            MSK["MSK Cluster"]:::stream
            MSK -->|"hosts"| TOPICS
        end

        %% --- Stream processing ---
        subgraph "EKS Cluster — Flink runtime"
            direction TB
            FLINKJM["Flink JobManager"]:::compute
            FLINKTM["Flink TaskManagers"]:::compute
            FLINKJOB["Flink Jobs\n(points_calc, fraud_detection)\n(state_backend: TBD/implicit)"]:::compute

            FLINKJM -->|"schedules"| FLINKTM
            FLINKJM -->|"runs"| FLINKJOB
            FLINKTM -->|"executes"| FLINKJOB
        end

        %% --- Serving store ---
        DDB["DynamoDB — Serving Store\n(current_points_balance, ledger, locks: logical)"]:::datastore

        %% --- API (CQRS read model) ---
        subgraph "API Layer — Query/Read Model"
            direction TB
            APIGW["API Gateway"]:::api
            LAMBDA["Lambda Functions\n(GetPointsBalance, GetTransactions, ...)"]:::compute
            APIGW -->|"invoke"| LAMBDA
        end

        %% --- Analytics / Data Lake ---
        subgraph "Historical / Analytics"
            direction TB
            EXPORT["Event Export\n(Kafka Connect / Flink / MSK Connect)"]:::integration
            S3["S3 — Event Archive / Data Lake"]:::datastore
            ATHENA["Athena — Analytics Query Engine"]:::analytics

            EXPORT -->|"write"| S3
            ATHENA -->|"query"| S3
        end

        %% --- Ops/Test client ---
        EC2CLIENT["EC2 Kafka Client\n(test/ops producer/consumer)"]:::compute
    end

    %% =========================
    %% Level 2: Deployment / IaC
    %% =========================
    subgraph "Deployment & IaC (Level 2 — Terraform-driven)"
        direction TB

        subgraph "Root Terraform Stack (orchestration)"
            direction TB
            TFROOT["Terraform Root Stack"]:::iac
            TFPROV["AWS Provider Config"]:::iac
            TFVARS["Root Variables"]:::iac
            TFOUT["Root Outputs"]:::iac

            TFPROV -->|"configures"| TFROOT
            TFVARS -->|"inputs"| TFROOT
            TFROOT -->|"emits"| TFOUT
        end

        subgraph "VPC Boundary (networking: subnets/SGs/NAT/IGW: TBD)"
            direction TB

            subgraph "Terraform Module: MSK Cluster"
                direction TB
                TFMSKMOD["msk_cluster module"]:::iac
                TFMSKMAIN["msk_cluster/main.tf"]:::iac
                TFMSKVARS["msk_cluster/variables.tf"]:::iac
                TFMSKOUT["msk_cluster/outputs.tf"]:::iac

                TFMSKMOD -->|"defined_by"| TFMSKMAIN
                TFMSKMOD -->|"inputs"| TFMSKVARS
                TFMSKMOD -->|"outputs"| TFMSKOUT
            end

            subgraph "Terraform Module: EC2 Client"
                direction TB
                TFEC2MOD["ec2_client module"]:::iac
                TFEC2MAIN["ec2_client/main.tf"]:::iac
                TFEC2VARS["ec2_client/variables.tf"]:::iac
                TFEC2OUT["ec2_client/outputs.tf"]:::iac

                TFEC2MOD -->|"defined_by"| TFEC2MAIN
                TFEC2MOD -->|"inputs"| TFEC2VARS
                TFEC2MOD -->|"outputs"| TFEC2OUT
            end

            IAM["IAM Roles/Policies\n(Flink IRSA/EKS nodes, Lambda exec, EC2 role)\n(MSK auth: IAM/SASL/SCRAM: TBD)"]:::security
        end

        DOCREADME["README — Intended Runtime Components"]:::doc
        DOCTOPICS["docs/kafka-topics.md — Topic Contracts"]:::doc
        DOCDRAWIO["docs/real-time-loyalty-platform.drawio — Baseline Diagram"]:::doc
    end

    %% =========================
    %% Cross-layer anchoring
    %% =========================
    PLATFORM -->|"runtime_components"| MSK
    PLATFORM -->|"runtime_components"| FLINKJOB
    PLATFORM -->|"runtime_components"| DDB
    PLATFORM -->|"runtime_components"| APIGW
    PLATFORM -->|"runtime_components"| S3

    TFROOT -->|"provisions"| TFMSKMOD
    TFROOT -->|"provisions"| TFEC2MOD

    TFMSKMOD -->|"creates"| MSK
    TFEC2MOD -->|"creates"| EC2CLIENT

    DOCTOPICS -->|"defines_contracts_for"| TOPICS
    DOCREADME -->|"documents"| PLATFORM
    DOCDRAWIO -->|"baseline_for"| PLATFORM

    %% =========================
    %% Data flows (explicit verbs)
    %% =========================
    POS -->|"produce"| TOPICS
    ECOM -->|"produce"| TOPICS
    PAY -->|"produce"| TOPICS
    PARTNER -->|"produce"| TOPICS
    EC2CLIENT -->|"produce/consume"| TOPICS

    TOPICS -->|"consume"| FLINKJOB
    FLINKJOB -->|"produce(derived)"| TOPICS
    FLINKJOB -->|"upsert/write"| DDB

    CUST -->|"https_request"| APIGW
    LAMBDA -->|"query_read_model"| DDB

    TOPICS -->|"export"| EXPORT
    ANALYST -->|"sql_queries"| ATHENA

    %% =========================
    %% Security / access (TBD but explicit boundary)
    %% =========================
    IAM -.->|"authz_for"| EC2CLIENT
    IAM -.->|"authz_for"| FLINKJOB
    IAM -.->|"authz_for"| LAMBDA
    IAM -.->|"authn_to_MSK(TBD)"| MSK

    %% =========================
    %% Click Events (from component_mapping)
    %% =========================
    click TFROOT "https://github.com/girishn/real-time-loyalty-platform/blob/main/infra/terraform/main.tf"
    click TFPROV "https://github.com/girishn/real-time-loyalty-platform/blob/main/infra/terraform/provider.tf"
    click TFVARS "https://github.com/girishn/real-time-loyalty-platform/blob/main/infra/terraform/variables.tf"
    click TFOUT "https://github.com/girishn/real-time-loyalty-platform/blob/main/infra/terraform/outputs.tf"

    click TFMSKMOD "https://github.com/girishn/real-time-loyalty-platform/tree/main/infra/terraform/msk_cluster/"
    click TFMSKMAIN "https://github.com/girishn/real-time-loyalty-platform/blob/main/infra/terraform/msk_cluster/main.tf"
    click TFMSKVARS "https://github.com/girishn/real-time-loyalty-platform/blob/main/infra/terraform/msk_cluster/variables.tf"
    click TFMSKOUT "https://github.com/girishn/real-time-loyalty-platform/blob/main/infra/terraform/msk_cluster/outputs.tf"

    click TFEC2MOD "https://github.com/girishn/real-time-loyalty-platform/tree/main/infra/terraform/ec2_client/"
    click TFEC2MAIN "https://github.com/girishn/real-time-loyalty-platform/blob/main/infra/terraform/ec2_client/main.tf"
    click TFEC2VARS "https://github.com/girishn/real-time-loyalty-platform/blob/main/infra/terraform/ec2_client/variables.tf"
    click TFEC2OUT "https://github.com/girishn/real-time-loyalty-platform/blob/main/infra/terraform/ec2_client/outputs.tf"

    click DOCTOPICS "https://github.com/girishn/real-time-loyalty-platform/blob/main/docs/kafka-topics.md"
    click DOCDRAWIO "https://github.com/girishn/real-time-loyalty-platform/blob/main/docs/real-time-loyalty-platform.drawio"
    click DOCREADME "https://github.com/girishn/real-time-loyalty-platform/blob/main/README.md"

    %% =========================
    %% Styles (color-coded)
    %% =========================
    classDef external fill:#E5E7EB,stroke:#6B7280,color:#111827,stroke-width:1px
    classDef boundary fill:#F3F4F6,stroke:#374151,color:#111827,stroke-width:2px
    classDef stream fill:#FDBA74,stroke:#9A3412,color:#111827,stroke-width:2px
    classDef contract fill:#FED7AA,stroke:#C2410C,color:#111827,stroke-width:1px
    classDef compute fill:#93C5FD,stroke:#1D4ED8,color:#0B1220,stroke-width:2px
    classDef datastore fill:#86EFAC,stroke:#166534,color:#052e16,stroke-width:2px
    classDef api fill:#60A5FA,stroke:#1E40AF,color:#0B1220,stroke-width:2px
    classDef analytics fill:#D8B4FE,stroke:#6D28D9,color:#1F1147,stroke-width:2px
    classDef integration fill:#FDE68A,stroke:#B45309,color:#111827,stroke-width:2px
    classDef iac fill:#FBCFE8,stroke:#9D174D,color:#3B0A1E,stroke-width:2px
    classDef security fill:#FCA5A5,stroke:#991B1B,color:#2B0B0B,stroke-width:2px
    classDef doc fill:#FEF3C7,stroke:#92400E,color:#111827,stroke-width:1px
```
