How logzio/sawmill works
Project Type and Purpose
• Sawmill is an open‑source Java library (Maven multi‑module project) for JSON document transformation and enrichment.
• It provides a configurable pipeline DSL to apply grok patterns, GeoIP lookups, user‑agent parsing, field/tag manipulation, filters, etc.
File Structure Analysis
• Top‑level modules:
• sawmill-core: the main library with pipeline engine, parsers, processors, condition evaluators, utilities, HTTP client, resources (grok patterns) and unit tests.
• sawmill-benchmark: a micro‑benchmarking app that uses sawmill-core to measure performance.
• Build/deploy: Maven wrapper, pom.xml, release scripts, GitHub Actions workflows.
• Key packages in sawmill-core:
• parser: turns JSON/HOCON configs into PipelineDefinition/ExecutionStepDefinition objects.
• core engine: Pipeline, PipelineExecutor, ExecutionStep, ProcessorExecutionStep, WatchedPipeline.
• processors: a catalog of Processor implementations (AddField, GrokProcessor, GeoIpProcessor, JsonProcessor, etc.).
• conditions: Condition implementations and factory registry.
• http: ExternalMappingsClient for fetching mappings over HTTP.
• utilities/mustache: templating and JSON utilities.
a. Main Components
• Configuration Parser: reads pipeline definition (JSON/HOCON) → builds in‑memory PipelineDefinition and ExecutionStepDefinition graph.
• Pipeline Engine: orchestrates ExecutionSteps and Processors.
• Processor Registry & Loader: discovers @ProcessorProvider implementations via annotations.
• Condition Registry & Loader: discovers @ConditionProvider implementations.
• Processors: individual transformation units (Grok, GeoIP, Filter, Add/Remove Field, etc.).
• External Services:
– GeoIP database (local .mmdb resource)
– ExternalMappingsClient (HTTP calls to mapping service)
• Benchmark App (sawmill-benchmark): standalone CLI tool invoking the core library.
b. Relationships and Data Flow
1. Input JSON document
→ Configuration Parser → PipelineDefinition
→ Pipeline Engine → for each document:
• For each ExecutionStep: evaluate Conditions (ConditionRegistry)
• Instantiate Processors (ProcessorRegistry)
• Apply transform to Doc object
→ Output transformed JSON
2. Processors may call utilities (Grok patterns, Mustache), look up GeoIP DB, or call ExternalMappingsClient over HTTP.
3. Benchmark module invokes the core pipeline and records metrics via PipelineExecutionMetricsTracker and PipelineExecutionMetricsMBean.
c. Architectural Patterns & Principles
• Plugin/Registry Pattern: dynamic discovery of processors and conditions via annotations and registries.
• Decorator/Chain of Responsibility: pipeline steps chained to process a document.
• Separation of Concerns: parsing, execution engine, processors, conditions, utilities are decoupled modules.
d. Key Technologies & Frameworks
• Java 8+; Maven multi‑module build
• Jackson/GSON for JSON handling (implied)
• Mustache templating
• MaxMind GeoIP2 library (.mmdb files)
• AhoCorasick for pattern matching
• JMX (PipelineExecutionMetricsMBean) for metrics
• JUnit/TestNG for tests