How markuskont/go-sigma-rule-engine works
Project Type and Purpose
• This project is a Go library (go-sigma-rule-engine) that implements:
• A Sigma rule parser (lexer + parser + AST)
• A runtime match engine (rule tree traversal)
• A CLI reference in cmd/ (per README, though not in the file tree)
• Example clients in examples/
• Its core purpose is to let developers embed Sigma rule parsing + matching into their own applications, so they can build lightweight log‑IDS systems without a heavyweight SIEM.
File Structure
Top‑level Go files (lexer.go, parser.go, nodes.go, tree.go, pattern.go, rule.go, ruleset.go, sigma.go, token.go, ident.go, errors.go) plus their _test.go counterparts implement the core Sigma engine:
• lexer.go / token.go / ident.go / errors.go → scanner & token definitions
• parser.go / nodes.go / tree.go → AST construction & rule tree
• pattern.go / rule.go → compiling AST into matchers
• ruleset.go → loading many rules into a collection, Ruleset, plus evaluation APIs
• sigma.go → top‑level entry point / facade
The examples/ directory shows sample CLI apps and how a user would feed events & rules to the engine.
Extract Key Architecture Insights from README
• There are two major versions of the engine (pkg/sigma/v1 and pkg/sigma/v2 historically), but the current root is the v2 rewrite.
• Core interfaces:
• Matcher (boolean Match(Event))
• Event (combines Keyworder and Selector)
• Usage flow:
1. sigma.NewRuleset reads rules from disk into in‑memory structures.
2. Client code loops over Ruleset.Rules or calls Ruleset.EvalAll to test incoming events.
Main Components & Their Interactions
a. Rule Ingestion Pipeline (Compile‑Time)
1. File I/O: YAML rule files
2. Lexical Analyzer (lexer.go) → tokens
3. Parser (parser.go) → parse tree (nodes.go)
4. AST → Pattern & Rule objects (pattern.go + rule.go)
b. Ruleset Manager (ruleset.go)
• Aggregates multiple Rule objects into a Ruleset
• Provides EvalAll(Event) and Rules slice
c. Runtime Match Engine
• Each Rule holds a tree of Matcher nodes
• For each incoming Event (client‑provided), the engine traverses the matcher tree
• Returns boolean match + detailed metadata
d. Client Application
• Implements Event interface (Keyworder + Selector) to adapt custom log formats
• Calls library APIs to load rules and evaluate events in real time
e. Optional CLI (cmd/sigma per README) and Examples
• Reference implementations to show typical usage patterns
Architectural Patterns & Design Principles
• Compiler pipeline for rule parsing (lexer → parser → AST → semantic → bytecode‑like matchers)
• Strategy pattern via the Matcher interface for extensible boolean expressions
• Dependency Injection for Event interface allowing any log/event format
• Separation of concerns: parsing vs. runtime matching vs. client code
Putting It All Together
• Interfaces (Matcher, Event) at appropriate boundaries
• Use notes or callouts to mention key files and packages