How airbnb/streamalert works
Main Components
a. Data Ingestion Layer:
• Kinesis Streams (shown in tf_kinesis_streams module)
• S3 Event inputs (tf_s3_events)
• CloudWatch Logs (tf_cloudwatch_logs_destination)
• CloudTrail (tf_cloudtrail)
b. Processing Layer:
• Classifier Lambda Function
• Handles parsing and normalization of incoming data
• Uses various parsers (JSON, CSV, KV, Syslog)
• Rules Engine Lambda Function
• Executes Python-based detection rules
• Processes normalized data against rule logic
c. Alert Management:
• Alert Merger Lambda Function
• Alert Processor Lambda Function
• Multiple output integrations (Slack, PagerDuty, Jira, etc.)
d. Supporting Services:
• Athena for historical searches
• DynamoDB for state management
• Lookup Tables (both DynamoDB and S3-based)
• Scheduled Query System
• Threat Intel Downloader
Component Grouping
Components into logical sections:
• Input Sources |
• ↓
• Data Processing |
• ↓
• Alert Management |
• ↓
• Output Destinations |
Additional Details
a. Infrastructure Layer:
• AWS Lambda containers
• IAM roles and permissions flow
• KMS encryption points
b. Monitoring & Management:
• CloudWatch Metrics
• CloudWatch Alarms
• Terraform management layer
Technical Details to Note
• Python-based rule execution environment
• Terraform-managed infrastructure
• Support for multiple input schemas
• Various output integration options
• Lookup table integration points