# logzio/sawmill architecture

> Architecture diagram of the GitHub repository logzio/sawmill, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/logzio/sawmill
- Repository: https://github.com/logzio/sawmill

## Overview

##### Project Type and Purpose
- Sawmill is an open‑source Java library (Maven multi‑module project) for JSON document transformation and enrichment.
- It provides a configurable pipeline DSL to apply grok patterns, GeoIP lookups, user‑agent parsing, field/tag manipulation, filters, etc.

##### File Structure Analysis
- Top‑level modules:
- sawmill-core: the main library with pipeline engine, parsers, processors, condition evaluators, utilities, HTTP client, resources (grok patterns) and unit tests.
- sawmill-benchmark: a micro‑benchmarking app that uses sawmill-core to measure performance.
- Build/deploy: Maven wrapper, pom.xml, release scripts, GitHub Actions workflows.
- Key packages in sawmill-core:
- parser: turns JSON/HOCON configs into PipelineDefinition/ExecutionStepDefinition objects.
- core engine: Pipeline, PipelineExecutor, ExecutionStep, ProcessorExecutionStep, WatchedPipeline.
- processors: a catalog of Processor implementations (AddField, GrokProcessor, GeoIpProcessor, JsonProcessor, etc.).
- conditions: Condition implementations and factory registry.
- http: ExternalMappingsClient for fetching mappings over HTTP.
- utilities/mustache: templating and JSON utilities.

a. Main Components
- Configuration Parser: reads pipeline definition (JSON/HOCON) → builds in‑memory PipelineDefinition and ExecutionStepDefinition graph.
- Pipeline Engine: orchestrates ExecutionSteps and Processors.
- Processor Registry & Loader: discovers @ProcessorProvider implementations via annotations.
- Condition Registry & Loader: discovers @ConditionProvider implementations.
- Processors: individual transformation units (Grok, GeoIP, Filter, Add/Remove Field, etc.).
- External Services:
– GeoIP database (local .mmdb resource)
– ExternalMappingsClient (HTTP calls to mapping service)
- Benchmark App (sawmill-benchmark): standalone CLI tool invoking the core library.
b. Relationships and Data Flow
1. Input JSON document
→ Configuration Parser → PipelineDefinition
→ Pipeline Engine → for each document:
- For each ExecutionStep: evaluate Conditions (ConditionRegistry)
- Instantiate Processors (ProcessorRegistry)
- Apply transform to Doc object
→ Output transformed JSON
2. Processors may call utilities (Grok patterns, Mustache), look up GeoIP DB, or call ExternalMappingsClient over HTTP.
3. Benchmark module invokes the core pipeline and records metrics via PipelineExecutionMetricsTracker and PipelineExecutionMetricsMBean.
c. Architectural Patterns & Principles
- Plugin/Registry Pattern: dynamic discovery of processors and conditions via annotations and registries.
- Decorator/Chain of Responsibility: pipeline steps chained to process a document.
- Separation of Concerns: parsing, execution engine, processors, conditions, utilities are decoupled modules.
d. Key Technologies & Frameworks
- Java 8+; Maven multi‑module build
- Jackson/GSON for JSON handling (implied)
- Mustache templating
- MaxMind GeoIP2 library (.mmdb files)
- AhoCorasick for pattern matching
- JMX (PipelineExecutionMetricsMBean) for metrics
- JUnit/TestNG for tests

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Input and Output
    InputJSON["Input JSON Document"]:::external
    OutputJSON["Output Transformed JSON"]:::external

    %% Parser Component
    subgraph "Configuration Parser"
        Parser["Configuration Parser"]:::core
    end

    %% Core Engine Components
    subgraph "Pipeline Engine"
        Pipeline["Pipeline"]:::core
        PipelineExecutor["PipelineExecutor"]:::core
        ExecutionStep["ExecutionStep"]:::core
    end

    %% Registry Components
    subgraph "Registries"
        ProcessorFactoryRegistry["ProcessorFactoryRegistry"]:::registry
        ProcessorFactoriesLoader["ProcessorFactoriesLoader"]:::registry
        ProcessorProviderAnno["@ProcessorProvider"]:::registry

        ConditionFactoryRegistry["ConditionFactoryRegistry"]:::registry
        ConditionalFactoriesLoader["ConditionalFactoriesLoader"]:::registry
        ConditionProviderAnno["@ConditionProvider"]:::registry
    end

    %% Plugin Modules
    subgraph "Plugin Modules"
        Processors["Processors"]:::plugin
        Conditions["Conditions"]:::plugin
    end

    %% Utilities & Templating
    subgraph "Utilities & Templating"
        GrokUtil["Grok Utility"]:::util
        JsonUtil["JsonUtils"]:::util
        MustacheFactories["Mustache Factories"]:::util
    end

    %% External Services
    subgraph "External Services"
        GeoIPDB["GeoIP Database (.mmdb)"]:::external
        ExternalMappingsClient["ExternalMappingsClient (HTTP)"]:::external
    end

    %% Benchmark App
    subgraph "Benchmark Application"
        BenchmarkMain["Main"]:::app
        BenchmarkLogic["SawmillBenchmark"]:::app
    end

    %% Data Flow and Relationships
    InputJSON -->|"parse config"| Parser
    Parser -->|"build PipelineDefinition"| Pipeline

    Pipeline -->|"execute steps"| PipelineExecutor
    PipelineExecutor -->|"iterate ExecutionStep"| ExecutionStep

    ExecutionStep -->|"evaluate condition"| ConditionFactoryRegistry
    ConditionFactoryRegistry -->|"find providers"| ConditionalFactoriesLoader
    ConditionalFactoriesLoader --> ConditionProviderAnno
    ExecutionStep -->|"instantiate processor"| ProcessorFactoryRegistry
    ProcessorFactoryRegistry -->|"load factories"| ProcessorFactoriesLoader
    ProcessorFactoriesLoader --> ProcessorProviderAnno

    ProcessorFactoryRegistry --> Processors
    ConditionalFactoriesLoader --> Conditions

    Processors -->|"apply transform"| ExecutionStep
    Processors -->|"use utilities"| GrokUtil
    Processors -->|"use utilities"| JsonUtil
    Processors -->|"use utilities"| MustacheFactories
    Processors -->|"lookup GeoIP"| GeoIPDB
    Processors -->|"fetch mapping over HTTP"| ExternalMappingsClient

    Conditions -->|"evaluate on document"| ExecutionStep

    PipelineExecutor -->|"return result"| OutputJSON

    BenchmarkMain -->|"invoke pipeline"| PipelineExecutor
    BenchmarkMain --> BenchmarkLogic

    %% Click Events for component mapping
    click Parser "https://github.com/logzio/sawmill/tree/master/sawmill-core/src/main/java/io/logz/sawmill/parser"
    click Pipeline "https://github.com/logzio/sawmill/blob/master/sawmill-core/src/main/java/io/logz/sawmill/Pipeline.java"
    click PipelineExecutor "https://github.com/logzio/sawmill/blob/master/sawmill-core/src/main/java/io/logz/sawmill/PipelineExecutor.java"
    click ExecutionStep "https://github.com/logzio/sawmill/blob/master/sawmill-core/src/main/java/io/logz/sawmill/ExecutionStep.java"
    click ProcessorFactoryRegistry "https://github.com/logzio/sawmill/blob/master/sawmill-core/src/main/java/io/logz/sawmill/ProcessorFactoryRegistry.java"
    click ProcessorFactoriesLoader "https://github.com/logzio/sawmill/blob/master/sawmill-core/src/main/java/io/logz/sawmill/ProcessorFactoriesLoader.java"
    click ProcessorProviderAnno "https://github.com/logzio/sawmill/blob/master/sawmill-core/src/main/java/io/logz/sawmill/annotations/ProcessorProvider.java"
    click ConditionFactoryRegistry "https://github.com/logzio/sawmill/blob/master/sawmill-core/src/main/java/io/logz/sawmill/ConditionFactoryRegistry.java"
    click ConditionalFactoriesLoader "https://github.com/logzio/sawmill/blob/master/sawmill-core/src/main/java/io/logz/sawmill/ConditionalFactoriesLoader.java"
    click ConditionProviderAnno "https://github.com/logzio/sawmill/blob/master/sawmill-core/src/main/java/io/logz/sawmill/annotations/ConditionProvider.java"
    click Processors "https://github.com/logzio/sawmill/tree/master/sawmill-core/src/main/java/io/logz/sawmill/processors"
    click Conditions "https://github.com/logzio/sawmill/tree/master/sawmill-core/src/main/java/io/logz/sawmill/conditions"
    click ExternalMappingsClient "https://github.com/logzio/sawmill/blob/master/sawmill-core/src/main/java/io/logz/sawmill/http/ExternalMappingsClient.java"
    click GeoIPDB "https://github.com/logzio/sawmill/blob/master/sawmill-core/src/test/resources/GeoIP2-City-Test.mmdb"
    click GrokUtil "https://github.com/logzio/sawmill/blob/master/sawmill-core/src/main/java/io/logz/sawmill/utilities/Grok.java"
    click JsonUtil "https://github.com/logzio/sawmill/blob/master/sawmill-core/src/main/java/io/logz/sawmill/utilities/JsonUtils.java"
    click MustacheFactories "https://github.com/logzio/sawmill/tree/master/sawmill-core/src/main/java/io/logz/sawmill/mustache/factories"
    click BenchmarkMain "https://github.com/logzio/sawmill/blob/master/sawmill-benchmark/src/main/java/io/logz/sawmill/benchmark/Main.java"
    click BenchmarkLogic "https://github.com/logzio/sawmill/blob/master/sawmill-benchmark/src/main/java/io/logz/sawmill/benchmark/SawmillBenchmark.java"

    %% Styles
    classDef core fill:#E3F2FD,stroke:#2196F3,color:#0D47A1;
    classDef registry fill:#FFF3E0,stroke:#FF9800,color:#E65100;
    classDef plugin fill:#E8F5E9,stroke:#4CAF50,color:#1B5E20;
    classDef util fill:#F3E5F5,stroke:#9C27B0,color:#4A148C;
    classDef external fill:#FFEBEE,stroke:#F44336,color:#B71C1C;
    classDef app fill:#ECEFF1,stroke:#607D8B,color:#263238;
```
