# markuskont/go-sigma-rule-engine architecture

> Architecture diagram of the GitHub repository markuskont/go-sigma-rule-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/markuskont/go-sigma-rule-engine
- Repository: https://github.com/markuskont/go-sigma-rule-engine

## Overview

##### Project Type and Purpose
- This project is a Go library (`go-sigma-rule-engine`) that implements:
- A Sigma rule parser (lexer + parser + AST)
- A runtime match engine (rule tree traversal)
- A CLI reference in `cmd/` (per README, though not in the file tree)
- Example clients in `examples/`
- Its core purpose is to let developers embed Sigma rule parsing + matching into their own applications, so they can build lightweight log‑IDS systems without a heavyweight SIEM.

##### File Structure
Top‑level Go files (lexer.go, parser.go, nodes.go, tree.go, pattern.go, rule.go, ruleset.go, sigma.go, token.go, ident.go, errors.go) plus their `_test.go` counterparts implement the core Sigma engine:
- lexer.go / token.go / ident.go / errors.go → scanner & token definitions
- parser.go / nodes.go / tree.go → AST construction & rule tree
- pattern.go / rule.go → compiling AST into matchers
- ruleset.go → loading many rules into a collection, `Ruleset`, plus evaluation APIs
- sigma.go → top‑level entry point / facade
The `examples/` directory shows sample CLI apps and how a user would feed events & rules to the engine.

##### Extract Key Architecture Insights from README
- There are two major versions of the engine (`pkg/sigma/v1` and `pkg/sigma/v2` historically), but the current root is the v2 rewrite.
- Core interfaces:
- `Matcher` (boolean `Match(Event)`)
- `Event` (combines `Keyworder` and `Selector`)
- Usage flow:
1. `sigma.NewRuleset` reads rules from disk into in‑memory structures.
2. Client code loops over `Ruleset.Rules` or calls `Ruleset.EvalAll` to test incoming events.

##### Main Components & Their Interactions
a. Rule Ingestion Pipeline (Compile‑Time)
1. File I/O: YAML rule files
2. Lexical Analyzer (lexer.go) → tokens
3. Parser (parser.go) → parse tree (nodes.go)
4. AST → `Pattern` & `Rule` objects (pattern.go + rule.go)
b. Ruleset Manager (ruleset.go)
- Aggregates multiple `Rule` objects into a `Ruleset`
- Provides `EvalAll(Event)` and `Rules` slice
c. Runtime Match Engine
- Each `Rule` holds a tree of `Matcher` nodes
- For each incoming `Event` (client‑provided), the engine traverses the matcher tree
- Returns boolean match + detailed metadata
d. Client Application
- Implements `Event` interface (Keyworder + Selector) to adapt custom log formats
- Calls library APIs to load rules and evaluate events in real time
e. Optional CLI (`cmd/sigma` per README) and Examples
- Reference implementations to show typical usage patterns

##### Architectural Patterns & Design Principles
- Compiler pipeline for rule parsing (lexer → parser → AST → semantic → bytecode‑like matchers)
- Strategy pattern via the `Matcher` interface for extensible boolean expressions
- Dependency Injection for `Event` interface allowing any log/event format
- Separation of concerns: parsing vs. runtime matching vs. client code

##### Putting It All Together
- Interfaces (`Matcher`, `Event`) at appropriate boundaries
- Use notes or callouts to mention key files and packages

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Compile-Time Pipeline
    subgraph "Compile-Time" 
        direction TB
        YAML["YAML Rule Files"]:::external
        Lex["Lexer"]:::parse
        Par["Parser & AST"]:::parse
        AST["AST"]:::parse
        Factory["Pattern & Rule Factory"]:::parse
        Manager["Ruleset Manager"]:::parse
    end

    %% Public API Facade
    subgraph "Library Public API" 
        direction TB
        Facade["Public API Facade\n(sigma.go)"]:::facade
    end

    %% Runtime Match Engine
    subgraph "Runtime Match Engine"
        direction TB
        EventSrc["Event Source\n(logs/events)"]:::external
        Adapter["Client Event Adapter\n(implements Event)"]:::runtime
        Engine["Match Engine"]:::runtime
        MatcherTree["Matcher Tree"]:::runtime
        Result["Match Result"]:::external
    end

    %% Example Clients
    subgraph "Example Clients" 
        direction TB
        CLI1["CLI: Parse Rules"]:::external
        CLI2["CLI: Simple Rule Mapping"]:::external
        CLI3["Example Streamer\n(Single-threaded)"]:::external
        CLI4["Example Streamer\n(Multi-threaded)"]:::external
    end

    %% Connections
    YAML -->|feeds| Lex
    Lex -->|tokens| Par
    Par -->|parse tree| AST
    AST -->|compile| Factory
    Factory -->|rules| Manager
    Manager -->|NewRuleset/EvalAll| Facade

    Facade -->|loads rules| Manager
    Facade -->|called by| Adapter

    EventSrc -->|events| Adapter
    Adapter -->|provides Event| Engine
    Engine -->|traverses| MatcherTree
    MatcherTree -->|results| Result
    Result -->|returned to| Adapter

    %% Examples usage
    CLI1 -->|uses| Facade
    CLI2 -->|uses| Facade
    CLI3 -->|uses| Facade
    CLI4 -->|uses| Facade

    %% Click Events
    click Lex "https://github.com/markuskont/go-sigma-rule-engine/blob/master/lexer.go"
    click Lex "https://github.com/markuskont/go-sigma-rule-engine/blob/master/token.go"
    click Lex "https://github.com/markuskont/go-sigma-rule-engine/blob/master/ident.go"
    click Lex "https://github.com/markuskont/go-sigma-rule-engine/blob/master/errors.go"
    click Par "https://github.com/markuskont/go-sigma-rule-engine/blob/master/parser.go"
    click Par "https://github.com/markuskont/go-sigma-rule-engine/blob/master/nodes.go"
    click Manager "https://github.com/markuskont/go-sigma-rule-engine/blob/master/ruleset.go"
    click Factory "https://github.com/markuskont/go-sigma-rule-engine/blob/master/pattern.go"
    click Factory "https://github.com/markuskont/go-sigma-rule-engine/blob/master/rule.go"
    click Facade "https://github.com/markuskont/go-sigma-rule-engine/blob/master/sigma.go"
    click CLI1 "https://github.com/markuskont/go-sigma-rule-engine/blob/master/examples/parse-rules/main.go"
    click CLI2 "https://github.com/markuskont/go-sigma-rule-engine/blob/master/examples/simple-rule-mapping/main.go"
    click CLI3 "https://github.com/markuskont/go-sigma-rule-engine/blob/master/examples/simple-streamer/main.go"
    click CLI4 "https://github.com/markuskont/go-sigma-rule-engine/blob/master/examples/threaded-streamer/main.go"

    %% Styles
    classDef parse fill:#eef7ff,stroke:#3b82f6,stroke-width:1px
    classDef runtime fill:#ecffec,stroke:#22c55e,stroke-width:1px
    classDef external fill:#fff7e6,stroke:#f97316,stroke-width:1px
    classDef facade fill:#ffe6e6,stroke:#ef4444,stroke-width:1px
```
