# jgilbert01/aws-lambda-stream architecture

> Architecture diagram of the GitHub repository jgilbert01/aws-lambda-stream, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jgilbert01/aws-lambda-stream
- Repository: https://github.com/jgilbert01/aws-lambda-stream

## Overview

##### Project Type and Purpose
- This is an **open-source library** for building AWS Lambda–based stream processors using Functional Reactive Programming (Highland.js).
- It provides **connectors**, **source builders**, **stream pipelines**, **flavors**, **sinks**, **fault handling**, **metrics**, and **utility** modules.
- It integrates with AWS streaming services (DynamoDB Streams, Kinesis, EventBridge, SNS, SQS, S3, CloudWatch, Firehose, Timestream, etc.).

##### Main Architectural Components
A. **AWS Event Sources**
- DynamoDB Stream
- Kinesis Stream
- EventBridge Bus
- SNS Topic
- SQS Queue
- S3 Notifications
- Scheduled (Cron) and Cognito streams
B. **Lambda Stream Processor (Core Library)**
- **Entry Point**: handler → calls `fromXxx(event)`
- **Stream Assembly**: `initialize` & `assemble` (pipelines fork/merge)
- **Pipelines & Flavors**: rule-driven or custom pipelines (in `src/pipelines`, `src/flavors`)
- **Filters & Mapping**: `src/filters`, mapping functions in user code
- **Fault Handling**: `src/faults`, `errors(faults)`, `flushFaults`
- **Metrics**: `src/metrics` (EMF, XRay integration, timers, utilization)
- **Utilities**: helpers for retry, rateLimit, batch, grouping, logging
C. **Connectors & Sinks**
- Thin wrappers around AWS SDK clients (`src/connectors`)
- Curryable sink functions (`src/sinks`) that integrate connectors into Highland streams (e.g., `update`, `publishToEventBridge`, `sendToSqs`, etc.)
D. **Templates & Examples**
- Project templates illustrating common architectures (`templates/…`)

##### Relationships & Data Flows
- **Inbound Flow**:
AWS Event Source → Lambda trigger → `fromXxx(event)` → Highland stream of `UnitOfWork`
- **Stream Processing Flow**:
‑ Fork into pipelines (`initialize` → fork/observe)
‑ Within each pipeline: `filter` → `map` (to UoW variables) → `through`(sink/flavor) → possible `parallel`, `batch`, `group` steps
‑ Merge pipelines → common tail: fault handling, metrics capture → `toPromise`
- **Outbound Flow**:
Sink steps invoke Connectors → AWS Services (DynamoDB, EventBridge, Kinesis, S3, SNS, SQS, Timestream, CloudWatch)
- **Error Flow**:
Pipeline error → catch via `errors(faults)` → buffer fault events → `flushFaults` → publish fault events (often to EventBridge or Dead-Letter)

##### Architectural Patterns & Design Principles
- **Functional Reactive Programming**: Highland.js streams model
- **CQRS / Event Sourcing**: Trigger functions publish domain events; listener functions materialize read models
- **Pipeline / Fork-Merge**: modular, testable pipelines assembled dynamically
- **Currying & Configuration**: sinks and flavors accept options (batchSize, parallelism, filters)
- **Immutability / UnitOfWork**: UoW objects flow through pipelines, enriched but never mutated
- **Backpressure & Parallelism**: `batch`, `parallel`, `rateLimit` for throughput control
- **Fault Isolation**: stream-circuit-breaker to handle poison records without losing entire batch
- **Metrics Instrumentation**: built-in CloudWatch EMF, XRay, custom metrics

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% AWS Event Sources
    subgraph "AWS Event Sources"
        style AWS_Event_Sources fill:none,stroke:none
        DynamoDB_Source["fromDynamodb()"]:::source
        Kinesis_Source["fromKinesis()"]:::source
        EventBridge_Source["fromEventBridge()"]:::source
        SNS_Source["fromSns()"]:::source
        SQS_Source["fromSqs()"]:::source
        S3_Source["fromS3()"]:::source
        Cron_Source["fromCron()"]:::source
        Cognito_Source["fromCognito()"]:::source
    end

    %% Lambda Processor Core
    subgraph "Lambda Stream Processor Core"
        style Core fill:none,stroke:none
        Handler["handler (entry)"]:::core
        Initialize["initialize & assemble"]:::core
        Pipelines["Pipelines & Utilities"]:::core
        Flavors["Flavors"]:::core
        Filters["Filters & Mapping"]:::core
        Faults["Fault Handling"]:::core
        Metrics["Metrics & Instrumentation"]:::core
        Utils["Utility Helpers"]:::core
    end

    %% Connectors & Sinks
    subgraph "Connectors & Sinks"
        style ConnectorsSinks fill:none,stroke:none
        Connectors["AWS SDK Connectors"]:::connector
        Sinks["Curryable Sinks"]:::sink
    end

    %% Outbound AWS Services
    subgraph "Outbound AWS Services"
        style Outbound fill:none,stroke:none
        DynamoDB["DynamoDB"]:::aws
        EventBridge["EventBridge"]:::aws
        Kinesis["Kinesis"]:::aws
        S3["S3"]:::aws
        SNS["SNS"]:::aws
        SQS["SQS"]:::aws
        Timestream["Timestream"]:::aws
        CloudWatch["CloudWatch"]:::aws
    end

    %% Data Flows
    DynamoDB_Source -->|event| Handler
    Kinesis_Source --> Handler
    EventBridge_Source --> Handler
    SNS_Source --> Handler
    SQS_Source --> Handler
    S3_Source --> Handler
    Cron_Source --> Handler
    Cognito_Source --> Handler

    Handler --> Initialize
    Initialize --> Pipelines
    Pipelines --> Filters
    Filters --> Flavors
    Flavors -->|fork/merge| Pipelines
    Pipelines --> Faults
    Pipelines --> Metrics
    Pipelines --> Utils

    Pipelines --> Connectors
    Connectors --> Sinks
    Sinks --> DynamoDB
    Sinks --> EventBridge
    Sinks --> Kinesis
    Sinks --> S3
    Sinks --> SNS
    Sinks --> SQS
    Sinks --> Timestream
    Sinks --> CloudWatch

    Faults -->|fault events| EventBridge
    Faults -->|dead-letter| SQS

    Metrics -.-> CloudWatch

    %% Click Events
    click DynamoDB_Source "https://github.com/jgilbert01/aws-lambda-stream/blob/master/src/from/dynamodb.js"
    click Kinesis_Source "https://github.com/jgilbert01/aws-lambda-stream/blob/master/src/from/kinesis.js"
    click EventBridge_Source "https://github.com/jgilbert01/aws-lambda-stream/blob/master/src/from/eventbridge.js"
    click SNS_Source "https://github.com/jgilbert01/aws-lambda-stream/blob/master/src/from/sns.js"
    click SQS_Source "https://github.com/jgilbert01/aws-lambda-stream/blob/master/src/from/sqs.js"
    click S3_Source "https://github.com/jgilbert01/aws-lambda-stream/blob/master/src/from/s3.js"
    click Cron_Source "https://github.com/jgilbert01/aws-lambda-stream/blob/master/src/from/cron.js"
    click Cognito_Source "https://github.com/jgilbert01/aws-lambda-stream/blob/master/src/from/cognito.js"
    click DynamoDB_Source "https://github.com/jgilbert01/aws-lambda-stream/tree/master/src/from"
    click Handler "https://github.com/jgilbert01/aws-lambda-stream/blob/master/src/index.js"
    click Initialize "https://github.com/jgilbert01/aws-lambda-stream/blob/master/src/pipelines/index.js"
    click Pipelines "https://github.com/jgilbert01/aws-lambda-stream/tree/master/src/pipelines"
    click Flavors "https://github.com/jgilbert01/aws-lambda-stream/tree/master/src/flavors"
    click Filters "https://github.com/jgilbert01/aws-lambda-stream/tree/master/src/filters"
    click Faults "https://github.com/jgilbert01/aws-lambda-stream/blob/master/src/faults/index.js"
    click Metrics "https://github.com/jgilbert01/aws-lambda-stream/tree/master/src/metrics"
    click Utils "https://github.com/jgilbert01/aws-lambda-stream/tree/master/src/utils"
    click Connectors "https://github.com/jgilbert01/aws-lambda-stream/tree/master/src/connectors"
    click Sinks "https://github.com/jgilbert01/aws-lambda-stream/tree/master/src/sinks"

    %% Styles
    classDef source fill:#adf7b6,stroke:#2d7f2d,shape:trapezoid
    classDef core fill:#a0c4ff,stroke:#003566,shape:rect
    classDef connector fill:#ffd6a5,stroke:#cc8400,shape:cylinder
    classDef sink fill:#dda0dd,stroke:#800080,shape:cloud
    classDef aws fill:#f0f0f0,stroke:#888888,shape:cloud
```
