# matanolabs/matano architecture

> Architecture diagram of the GitHub repository matanolabs/matano, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/matanolabs/matano
- Repository: https://github.com/matanolabs/matano

## Overview

##### Main System Components

a) Data Ingestion Layer
- Various input sources (50+ supported log sources)
- S3 buckets and SQS queues as entry points
- Log puller components for different services (found in lib/rust/log_puller/src/pullers/)

b) Processing Layer
- Data Transformer (lib/rust/transformer)
- Data Batcher (lib/rust/data_batcher)
- Lake Writer (lib/rust/lake_writer)
- VRL Processing Pipeline

c) Storage Layer
- Apache Iceberg Tables
- Schema management system (data/ecs_iceberg_schema.json)
- Relationship with AWS Athena

d) Detection & Alerting Layer
- Detection Engine (Python-based)
- Alert Writer (lib/rust/alert_writer)
- Alert Forwarder (lib/rust/alert_forwarder)
- Remote Cache System for detections

##### Component Relationships & Data Flow

- Raw log ingestion flow from sources → S3/SQS
- Data transformation path through the processing layer
- Write path to Iceberg tables
- Detection execution flow
- Alert delivery to destinations (Slack, SNS)

- Blue: AWS services (S3, SQS, Lambda)
- Green: Data processing components
- Yellow: Storage layer
- Red: Detection & alerting components
- Purple: External integrations

##### Key Technical Elements to Highlight

a) Infrastructure Components:
- AWS Lambda functions for serverless processing
- S3 buckets for data storage
- SQS queues for message handling
- SNS topics for alert distribution

b) Data Processing:
- VRL transformation pipeline
- ECS schema compliance
- Iceberg table format

c) Security Features:
- Detection-as-code framework
- Real-time alert processing
- Enrichment capabilities

##### Integration Points

Connections to:
- External log sources (AWS services, Cloudflare, Okta, etc.)
- Query engines (Athena, Snowflake, Spark, Trino)
- Alert destinations (Slack, Email)

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Data Ingestion Layer"
        direction TB
        Sources["50+ Log Sources"]
        S3["AWS S3 Buckets"]:::aws
        SQS["AWS SQS Queues"]:::aws
        LogPullers["Log Pullers"]:::processor
    end

    subgraph "Processing Layer"
        direction TB
        DataTransformer["Data Transformer"]:::processor
        DataBatcher["Data Batcher"]:::processor
        LakeWriter["Lake Writer"]:::processor
        VRLPipeline["VRL Processing Pipeline"]:::processor
    end

    subgraph "Storage Layer"
        direction TB
        IcebergTables["Apache Iceberg Tables"]:::storage
        SchemaManager["Schema Management"]:::storage
        Athena["AWS Athena"]:::aws
    end

    subgraph "Detection & Alerting Layer"
        direction TB
        DetectionEngine["Detection Engine"]:::security
        AlertWriter["Alert Writer"]:::security
        AlertForwarder["Alert Forwarder"]:::security
        RemoteCache["Remote Cache"]:::security
    end

    subgraph "External Integrations"
        direction TB
        QueryEngines["Query Engines\n(Snowflake, Spark, Trino)"]
        AlertDestinations["Alert Destinations\n(Slack, SNS)"]
    end

    %% Data Flow Connections
    Sources --> S3
    Sources --> SQS
    S3 --> LogPullers
    SQS --> LogPullers
    LogPullers --> DataTransformer
    DataTransformer --> DataBatcher
    DataBatcher --> LakeWriter
    VRLPipeline --> DataTransformer
    LakeWriter --> IcebergTables
    IcebergTables --> SchemaManager
    IcebergTables --> Athena
    IcebergTables --> DetectionEngine
    DetectionEngine --> AlertWriter
    AlertWriter --> AlertForwarder
    AlertForwarder --> AlertDestinations
    DetectionEngine --> RemoteCache
    IcebergTables --> QueryEngines

    %% Click Events
    click LogPullers "https://github.com/matanolabs/matano/tree/main/lib/rust/log_puller/src/pullers/"
    click DataTransformer "https://github.com/matanolabs/matano/tree/main/lib/rust/transformer/"
    click DataBatcher "https://github.com/matanolabs/matano/tree/main/lib/rust/data_batcher/"
    click LakeWriter "https://github.com/matanolabs/matano/tree/main/lib/rust/lake_writer/"
    click AlertWriter "https://github.com/matanolabs/matano/tree/main/lib/rust/alert_writer/"
    click AlertForwarder "https://github.com/matanolabs/matano/tree/main/lib/rust/alert_forwarder/"
    click DetectionEngine "https://github.com/matanolabs/matano/tree/main/lib/python/matano_detection/"
    click SchemaManager "https://github.com/matanolabs/matano/blob/main/data/ecs_iceberg_schema.json"
    click VRLPipeline "https://github.com/matanolabs/matano/blob/main/lib/rust/shared/src/vrl_util.rs"
    click AlertDestinations "https://github.com/matanolabs/matano/tree/main/example/integrations/destinations/"
    click IcebergTables "https://github.com/matanolabs/matano/tree/main/lib/java/matano/iceberg_main/"

    %% Styling
    classDef aws fill:#2196F3,stroke:#fff,stroke-width:2px,color:#fff
    classDef processor fill:#4CAF50,stroke:#fff,stroke-width:2px,color:#fff
    classDef storage fill:#FFC107,stroke:#fff,stroke-width:2px,color:#000
    classDef security fill:#F44336,stroke:#fff,stroke-width:2px,color:#fff
    classDef integration fill:#9C27B0,stroke:#fff,stroke-width:2px,color:#fff

    %% Legend
    subgraph Legend
        direction TB
        L1["AWS Services"]:::aws
        L2["Processing Components"]:::processor
        L3["Storage Components"]:::storage
        L4["Security Components"]:::security
        L5["External Integrations"]:::integration
    end
```
