How amzn/amazon-marketing-stream-examples works
Project Type and Purpose
• This is an open-source reference implementation in Python that combines a CLI tool (amz_stream_cli) with AWS CDK infrastructure code (amz_stream_infra and app.py) to:
– Provision AWS resources for consuming Amazon Marketing Stream data via SQS or Firehose.
– Provide Lambda functions for message fan-out, subscription confirmation, and batch processing.
– Offer a CLI for managing stream dataset subscriptions.
File-Structure Analysis
• Top level
– app.py & cdk.json: Entry point for CDK to synthesize/deploy stacks.
– stream_infrastructure_config.yml: Configuration for stack rollout behavior.
• amz_stream_cli/
– __main__.py, cli.py: Click-based CLI commands (create, get, list, update).
– stream_api.py: HTTP client for Amazon Marketing Stream subscription API.
• amz_stream_infra/
– infra_rollout.py: Orchestrates deployment of multiple dataset stacks.
– stack_definitions.py & stack_definitions_firehose.py: Define AWS CDK constructs for SQS and Firehose deployments respectively.
• lambda/
– aws_clients.py: Shared AWS SDK clients.
– stream_fanout_lambda.py: Fan-out incoming messages to SNS or confirmation queue.
– subscription_confirmation_lambda.py: Confirms SQS subscriptions.
– sqs_consuming_lambda.py & batch.py: Helpers for batch processing and SQS consumption.
• tests/: Unit tests for CDK stack definitions.
Insights from README
• Two delivery modes: SQS (default) and Firehose (via CDK context flag).
• SQS architecture: IngressQueue → Fan-out Lambda → (SubscriptionConfirmationQueue + Lambda) and DataTopic → Firehose → S3 Bucket.
• Firehose architecture: Direct Firehose → S3 Bucket with IAM roles.
• CLI manages the subscription lifecycle using Amazon Ads API credentials.
a. Main Components
1. CLI Layer (amz_stream_cli) – interacts with Amazon Marketing Stream REST API.
2. CDK Control Plane (app.py + amz_stream_infra) – synthesizes and deploys AWS infrastructure.
3. AWS Infrastructure:
• SQS Ingress Queues + Dead-Letter Queues
• Lambdas: StreamFanoutLambda, SubscriptionConfirmationLambda, SqsConsumingLambda
• SNS Topic for data forwarding
• Kinesis Data Firehose Delivery Stream
• S3 Landing Zone Bucket
• IAM Roles (FirehoseSubscriberRole & FirehoseSubscriptionRole)
4. External Services:
• Amazon Marketing Stream API (source of messages)
• AWS Services (SQS, Lambda, SNS, Firehose, S3, IAM)
b. Relationships & Data Flows
– CLI → Amazon Marketing Stream API (subscription calls).
– Marketing Stream → SQS IngressQueue (subscription target) or Firehose Delivery Stream.
– SQS → triggers StreamFanoutLambda:
· If message is subscription-confirmation → push to ConfirmationQueue → SubscriptionConfirmationLambda → call Stream API.
· If data message → publish to SNS Topic → Firehose → S3.
– In Firehose mode: Marketing Stream → Firehose → S3 (no Lambdas).
c. Architectural Patterns & Principles
– Cloud-native event-driven architecture
– Infrastructure as Code (AWS CDK)
– Separation of concerns: CLI vs infra vs runtime Lambdas
– Dead-letter queues for resilience
d. Key Technologies & Frameworks
– Python 3.7+, AWS CDK v2
– Click (CLI framework)
– Boto3 (AWS SDK)
– AWS Services: SQS, SNS, Lambda, Firehose, S3, IAM
– PyTest for unit testing CDK stacks