How matanolabs/matano works
Main System Components
a) Data Ingestion Layer
• Various input sources (50+ supported log sources)
• S3 buckets and SQS queues as entry points
• Log puller components for different services (found in lib/rust/log_puller/src/pullers/)
b) Processing Layer
• Data Transformer (lib/rust/transformer)
• Data Batcher (lib/rust/data_batcher)
• Lake Writer (lib/rust/lake_writer)
• VRL Processing Pipeline
c) Storage Layer
• Apache Iceberg Tables
• Schema management system (data/ecs_iceberg_schema.json)
• Relationship with AWS Athena
d) Detection & Alerting Layer
• Detection Engine (Python-based)
• Alert Writer (lib/rust/alert_writer)
• Alert Forwarder (lib/rust/alert_forwarder)
• Remote Cache System for detections
Component Relationships & Data Flow
• Raw log ingestion flow from sources → S3/SQS
• Data transformation path through the processing layer
• Write path to Iceberg tables
• Detection execution flow
• Alert delivery to destinations (Slack, SNS)
• Blue: AWS services (S3, SQS, Lambda)
• Green: Data processing components
• Yellow: Storage layer
• Red: Detection & alerting components
• Purple: External integrations
Key Technical Elements to Highlight
a) Infrastructure Components:
• AWS Lambda functions for serverless processing
• S3 buckets for data storage
• SQS queues for message handling
• SNS topics for alert distribution
b) Data Processing:
• VRL transformation pipeline
• ECS schema compliance
• Iceberg table format
c) Security Features:
• Detection-as-code framework
• Real-time alert processing
• Enrichment capabilities
Integration Points
Connections to:
• External log sources (AWS services, Cloudflare, Okta, etc.)
• Query engines (Athena, Snowflake, Spark, Trino)
• Alert destinations (Slack, Email)