How jgilbert01/aws-lambda-stream works
Project Type and Purpose
• This is an open-source library for building AWS Lambda–based stream processors using Functional Reactive Programming (Highland.js).
• It provides connectors, source builders, stream pipelines, flavors, sinks, fault handling, metrics, and utility modules.
• It integrates with AWS streaming services (DynamoDB Streams, Kinesis, EventBridge, SNS, SQS, S3, CloudWatch, Firehose, Timestream, etc.).
Main Architectural Components
A. AWS Event Sources
• DynamoDB Stream
• Kinesis Stream
• EventBridge Bus
• SNS Topic
• SQS Queue
• S3 Notifications
• Scheduled (Cron) and Cognito streams
B. Lambda Stream Processor (Core Library)
• Entry Point: handler → calls fromXxx(event)
• Stream Assembly: initialize & assemble (pipelines fork/merge)
• Pipelines & Flavors: rule-driven or custom pipelines (in src/pipelines, src/flavors)
• Filters & Mapping: src/filters, mapping functions in user code
• Fault Handling: src/faults, errors(faults), flushFaults
• Metrics: src/metrics (EMF, XRay integration, timers, utilization)
• Utilities: helpers for retry, rateLimit, batch, grouping, logging
C. Connectors & Sinks
• Thin wrappers around AWS SDK clients (src/connectors)
• Curryable sink functions (src/sinks) that integrate connectors into Highland streams (e.g., update, publishToEventBridge, sendToSqs, etc.)
D. Templates & Examples
• Project templates illustrating common architectures (templates/…)
Relationships & Data Flows
• Inbound Flow:
AWS Event Source → Lambda trigger → fromXxx(event) → Highland stream of UnitOfWork
• Stream Processing Flow:
‑ Fork into pipelines (initialize → fork/observe)
‑ Within each pipeline: filter → map (to UoW variables) → through(sink/flavor) → possible parallel, batch, group steps
‑ Merge pipelines → common tail: fault handling, metrics capture → toPromise
• Outbound Flow:
Sink steps invoke Connectors → AWS Services (DynamoDB, EventBridge, Kinesis, S3, SNS, SQS, Timestream, CloudWatch)
• Error Flow:
Pipeline error → catch via errors(faults) → buffer fault events → flushFaults → publish fault events (often to EventBridge or Dead-Letter)
Architectural Patterns & Design Principles
• Functional Reactive Programming: Highland.js streams model
• CQRS / Event Sourcing: Trigger functions publish domain events; listener functions materialize read models
• Pipeline / Fork-Merge: modular, testable pipelines assembled dynamically
• Currying & Configuration: sinks and flavors accept options (batchSize, parallelism, filters)
• Immutability / UnitOfWork: UoW objects flow through pipelines, enriched but never mutated
• Backpressure & Parallelism: batch, parallel, rateLimit for throughput control
• Fault Isolation: stream-circuit-breaker to handle poison records without losing entire batch
• Metrics Instrumentation: built-in CloudWatch EMF, XRay, custom metrics