# airbnb/streamalert architecture

> Architecture diagram of the GitHub repository airbnb/streamalert, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/airbnb/streamalert
- Repository: https://github.com/airbnb/streamalert

## Overview

##### Main Components

a. Data Ingestion Layer:
- Kinesis Streams (shown in tf_kinesis_streams module)
- S3 Event inputs (tf_s3_events)
- CloudWatch Logs (tf_cloudwatch_logs_destination)
- CloudTrail (tf_cloudtrail)

b. Processing Layer:
- Classifier Lambda Function
- Handles parsing and normalization of incoming data
- Uses various parsers (JSON, CSV, KV, Syslog)
- Rules Engine Lambda Function
- Executes Python-based detection rules
- Processes normalized data against rule logic

c. Alert Management:
- Alert Merger Lambda Function
- Alert Processor Lambda Function
- Multiple output integrations (Slack, PagerDuty, Jira, etc.)

d. Supporting Services:
- Athena for historical searches
- DynamoDB for state management
- Lookup Tables (both DynamoDB and S3-based)
- Scheduled Query System
- Threat Intel Downloader

##### Component Grouping
Components into logical sections:

- Input Sources     |

- ↓

- Data Processing    |

- ↓

- Alert Management    |

- ↓

- Output Destinations  |

##### Additional Details

a. Infrastructure Layer:
- AWS Lambda containers
- IAM roles and permissions flow
- KMS encryption points

b. Monitoring & Management:
- CloudWatch Metrics
- CloudWatch Alarms
- Terraform management layer

##### Technical Details to Note

- Python-based rule execution environment
- Terraform-managed infrastructure
- Support for multiple input schemas
- Various output integration options
- Lookup table integration points

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles
    classDef aws fill:#FF9900,stroke:#232F3E,color:white
    classDef input fill:#4CAF50,stroke:#1B5E20,color:white
    classDef processing fill:#FFC107,stroke:#FF8F00,color:black
    classDef alert fill:#F44336,stroke:#B71C1C,color:white
    classDef support fill:#9C27B0,stroke:#4A148C,color:white
    classDef integration fill:#2196F3,stroke:#0D47A1,color:white

    %% Input Sources
    subgraph InputSources
        KS["Kinesis Streams"]:::aws
        S3["S3 Events"]:::aws
        CWL["CloudWatch Logs"]:::aws
        CT["CloudTrail"]:::aws
    end

    %% Processing Layer
    subgraph Processing
        CL["Classifier Lambda"]:::processing
        RE["Rules Engine"]:::processing
        Schemas["Schema Definitions"]:::processing
        Rules["Rule Definitions"]:::processing
    end

    %% Alert Management
    subgraph AlertManagement
        AM["Alert Merger"]:::alert
        AP["Alert Processor"]:::alert
    end

    %% Supporting Services
    subgraph SupportingServices
        Athena["Athena"]:::support
        DynamoDB["DynamoDB"]:::support
        LT["Lookup Tables"]:::support
        SQS["Scheduled Query System"]:::support
        TID["Threat Intel Downloader"]:::support
    end

    %% Output Integrations
    subgraph Outputs
        Slack["Slack"]:::integration
        PagerDuty["PagerDuty"]:::integration
        Jira["Jira"]:::integration
    end

    %% Relationships
    KS --> CL
    S3 --> CL
    CWL --> CL
    CT --> CL
    
    CL --> RE
    Schemas --> CL
    Rules --> RE
    
    RE --> AM
    AM --> AP
    
    AP --> Slack
    AP --> PagerDuty
    AP --> Jira
    
    LT --> RE
    TID --> RE
    SQS --> RE
    
    RE --> Athena
    RE --> DynamoDB

    %% Click Events
    click KS "https://github.com/airbnb/streamalert/tree/master/streamalert_cli/_infrastructure/modules/tf_kinesis_streams/"
    click S3 "https://github.com/airbnb/streamalert/tree/master/streamalert_cli/_infrastructure/modules/tf_s3_events/"
    click CWL "https://github.com/airbnb/streamalert/tree/master/streamalert_cli/_infrastructure/modules/tf_cloudwatch_logs_destination/"
    click CT "https://github.com/airbnb/streamalert/tree/master/streamalert_cli/_infrastructure/modules/tf_cloudtrail/"
    click CL "https://github.com/airbnb/streamalert/tree/master/streamalert/classifier/"
    click RE "https://github.com/airbnb/streamalert/tree/master/streamalert/rules_engine/"
    click AM "https://github.com/airbnb/streamalert/tree/master/streamalert/alert_merger/"
    click AP "https://github.com/airbnb/streamalert/tree/master/streamalert/alert_processor/"
    click Athena "https://github.com/airbnb/streamalert/tree/master/streamalert_cli/_infrastructure/modules/tf_athena/"
    click LT "https://github.com/airbnb/streamalert/tree/master/streamalert/shared/lookup_tables/"
    click SQS "https://github.com/airbnb/streamalert/tree/master/streamalert/scheduled_queries/"
    click TID "https://github.com/airbnb/streamalert/tree/master/streamalert/threat_intel_downloader/"
    click Outputs "https://github.com/airbnb/streamalert/tree/master/streamalert/alert_processor/outputs/"
    click Rules "https://github.com/airbnb/streamalert/tree/master/rules/community/"
    click Schemas "https://github.com/airbnb/streamalert/tree/master/conf/schemas/"

    %% Legend
    subgraph Legend
        L1["AWS Service"]:::aws
        L2["Input Source"]:::input
        L3["Processing"]:::processing
        L4["Alert"]:::alert
        L5["Support"]:::support
        L6["Integration"]:::integration
    end
```
