# amzn/amazon-marketing-stream-examples architecture

> Architecture diagram of the GitHub repository amzn/amazon-marketing-stream-examples, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/amzn/amazon-marketing-stream-examples
- Repository: https://github.com/amzn/amazon-marketing-stream-examples

## Overview

##### Project Type and Purpose
- This is an open-source reference implementation in Python that combines a CLI tool (amz_stream_cli) with AWS CDK infrastructure code (amz_stream_infra and app.py) to:
– Provision AWS resources for consuming Amazon Marketing Stream data via SQS or Firehose.
– Provide Lambda functions for message fan-out, subscription confirmation, and batch processing.
– Offer a CLI for managing stream dataset subscriptions.

##### File-Structure Analysis
- Top level
– app.py & cdk.json: Entry point for CDK to synthesize/deploy stacks.
– stream_infrastructure_config.yml: Configuration for stack rollout behavior.
- amz_stream_cli/
– __main__.py, cli.py: Click-based CLI commands (create, get, list, update).
– stream_api.py: HTTP client for Amazon Marketing Stream subscription API.
- amz_stream_infra/
– infra_rollout.py: Orchestrates deployment of multiple dataset stacks.
– stack_definitions.py & stack_definitions_firehose.py: Define AWS CDK constructs for SQS and Firehose deployments respectively.
- lambda/
– aws_clients.py: Shared AWS SDK clients.
– stream_fanout_lambda.py: Fan-out incoming messages to SNS or confirmation queue.
– subscription_confirmation_lambda.py: Confirms SQS subscriptions.
– sqs_consuming_lambda.py & batch.py: Helpers for batch processing and SQS consumption.
- tests/: Unit tests for CDK stack definitions.

##### Insights from README
- Two delivery modes: SQS (default) and Firehose (via CDK context flag).
- SQS architecture: IngressQueue → Fan-out Lambda → (SubscriptionConfirmationQueue + Lambda) and DataTopic → Firehose → S3 Bucket.
- Firehose architecture: Direct Firehose → S3 Bucket with IAM roles.
- CLI manages the subscription lifecycle using Amazon Ads API credentials.

a. Main Components
1. CLI Layer (amz_stream_cli) – interacts with Amazon Marketing Stream REST API.
2. CDK Control Plane (app.py + amz_stream_infra) – synthesizes and deploys AWS infrastructure.
3. AWS Infrastructure:
- SQS Ingress Queues + Dead-Letter Queues
- Lambdas: StreamFanoutLambda, SubscriptionConfirmationLambda, SqsConsumingLambda
- SNS Topic for data forwarding
- Kinesis Data Firehose Delivery Stream
- S3 Landing Zone Bucket
- IAM Roles (FirehoseSubscriberRole & FirehoseSubscriptionRole)
4. External Services:
- Amazon Marketing Stream API (source of messages)
- AWS Services (SQS, Lambda, SNS, Firehose, S3, IAM)
b. Relationships & Data Flows
– CLI → Amazon Marketing Stream API (subscription calls).
– Marketing Stream → SQS IngressQueue (subscription target) or Firehose Delivery Stream.
– SQS → triggers StreamFanoutLambda:
· If message is subscription-confirmation → push to ConfirmationQueue → SubscriptionConfirmationLambda → call Stream API.
· If data message → publish to SNS Topic → Firehose → S3.
– In Firehose mode: Marketing Stream → Firehose → S3 (no Lambdas).
c. Architectural Patterns & Principles
– Cloud-native event-driven architecture
– Infrastructure as Code (AWS CDK)
– Separation of concerns: CLI vs infra vs runtime Lambdas
– Dead-letter queues for resilience
d. Key Technologies & Frameworks
– Python 3.7+, AWS CDK v2
– Click (CLI framework)
– Boto3 (AWS SDK)
– AWS Services: SQS, SNS, Lambda, Firehose, S3, IAM
– PyTest for unit testing CDK stacks

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Systems
    subgraph "External" 
        ExtAPI["Amazon Marketing Stream API"]:::gray
    end

    %% Control Plane
    subgraph "Control Plane"
        CLI_main["__main__.py"]:::blue
        CLI_cli["cli.py"]:::blue
        CLI_api["stream_api.py"]:::blue
        CDK_entry["app.py / cdk.json"]:::blue
        Infra_rollout["infra_rollout.py"]:::blue
        SQS_defs["stack_definitions.py"]:::blue
        FH_defs["stack_definitions_firehose.py"]:::blue
        Config["stream_infrastructure_config.yml"]:::blue
        Tests["test_amz_stream_wings_stack.py"]:::blue
    end

    %% Data Plane (AWS)
    subgraph "Data Plane (AWS)"
        subgraph "SQS Delivery Mode" 
            IngressQ["Ingress Queue (SQS)"]:::orange
            DLQ["Dead-Letter Queue"]:::orange
            FanoutLambda["StreamFanoutLambda"]:::green
            ConfQueue["SubscriptionConfirmationQueue"]:::orange
            ConfirmLambda["SubscriptionConfirmationLambda"]:::green
            SNSTopic["SNS Topic"]:::orange
            Firehose["Kinesis Data Firehose"]:::orange
            S3["S3 Landing Zone Bucket"]:::brown
            FirehoseRole["IAM Roles"]:::blue
        end
        subgraph "Firehose Delivery Mode"
            FH_Direct["Firehose Delivery Stream"]:::orange
            FH_S3["S3 Landing Zone Bucket"]:::brown
            FH_Role["IAM Roles"]:::blue
        end
        SharedClients["aws_clients.py"]:::green
        ConsumingLambda["sqs_consuming_lambda.py & batch.py"]:::green
    end

    %% Relationships
    CLI_main --> CLI_cli
    CLI_cli --> CLI_api
    CLI_cli -->|"reads config"| Config
    CLI_cli -->|"invoke deploy"| CDK_entry
    CDK_entry --> Infra_rollout
    Infra_rollout --> SQS_defs
    Infra_rollout --> FH_defs
    INFCRM{{"Data Plane Deployment"}} 
    Infra_rollout --> INFCRM

    %% Subscription API
    CLI_api -->|"Subscribe API call"| ExtAPI

    %% Message Ingress Paths
    ExtAPI -->|"Ingest message"| IngressQ
    ExtAPI -->|"Ingest message"| FH_Direct

    IngressQ -->|"trigger"| FanoutLambda
    FanoutLambda -->|"confirm message"| ConfQueue
    FanoutLambda -->|"data message"| SNSTopic

    ConfQueue -->|"trigger"| ConfirmLambda
    ConfirmLambda -->|"Confirm subscription API call"| ExtAPI

    SNSTopic -->|"deliver to Firehose"| Firehose
    Firehose -->|"store object"| S3

    FH_Direct -->|"deliver to S3"| FH_S3
    FH_Direct -->|"assume role"| FH_Role

    %% Shared Lambda clients and batch
    SharedClients -.-> FanoutLambda
    SharedClients -.-> ConfirmLambda
    SharedClients -.-> ConsumingLambda
    ConsumingLambda --> IngressQ

    %% Click Events
    click CLI_main "https://github.com/amzn/amazon-marketing-stream-examples/blob/main/amz_stream_cli/__main__.py"
    click CLI_cli "https://github.com/amzn/amazon-marketing-stream-examples/blob/main/amz_stream_cli/cli.py"
    click CLI_api "https://github.com/amzn/amazon-marketing-stream-examples/blob/main/amz_stream_cli/stream_api.py"
    click CDK_entry "https://github.com/amzn/amazon-marketing-stream-examples/blob/main/app.py"
    click CDK_entry "https://github.com/amzn/amazon-marketing-stream-examples/blob/main/cdk.json"
    click Infra_rollout "https://github.com/amzn/amazon-marketing-stream-examples/blob/main/amz_stream_infra/infra_rollout.py"
    click SQS_defs "https://github.com/amzn/amazon-marketing-stream-examples/blob/main/amz_stream_infra/stack_definitions.py"
    click FH_defs "https://github.com/amzn/amazon-marketing-stream-examples/blob/main/amz_stream_infra/stack_definitions_firehose.py"
    click Config "https://github.com/amzn/amazon-marketing-stream-examples/blob/main/stream_infrastructure_config.yml"
    click SharedClients "https://github.com/amzn/amazon-marketing-stream-examples/blob/main/lambda/aws_clients.py"
    click FanoutLambda "https://github.com/amzn/amazon-marketing-stream-examples/blob/main/lambda/stream_fanout_lambda.py"
    click ConfirmLambda "https://github.com/amzn/amazon-marketing-stream-examples/blob/main/lambda/subscription_confirmation_lambda.py"
    click ConsumingLambda "https://github.com/amzn/amazon-marketing-stream-examples/blob/main/lambda/sqs_consuming_lambda.py"
    click ConsumingLambda "https://github.com/amzn/amazon-marketing-stream-examples/blob/main/lambda/batch.py"
    click Tests "https://github.com/amzn/amazon-marketing-stream-examples/blob/main/tests/unit/test_amz_stream_wings_stack.py"

    %% Styles
    classDef blue fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef green fill:#90EE90,stroke:#333,stroke-width:1px
    classDef orange fill:#FFA500,stroke:#333,stroke-width:1px
    classDef brown fill:#D2B48C,stroke:#333,stroke-width:1px
    classDef gray fill:#D3D3D3,stroke:#333,stroke-width:1px
```
