How elastic/detection-rules works
This project is a Python‐based “Detection Rules” repository designed to build, validate, test, package, and integrate detection rules into Elastic Security’s Detection Engine (Kibana). The project is not a typical full-stack web application but a highly modular library and CLI tool that supports “detections as code” with multiple supporting libraries (e.g. for Kibana API interactions and KQL parsing) and extensive testing and workflow automation.
Main Components
• CLI Interface & Entry Points: There is a “detection_rules” command line interface (see the “CLI.md” and the __main__.py files in relevant modules) that developers and automation use to create, validate, import, search, and test rules.
• Rule Core Module: The detection_rules/ directory contains the core Python module. It is responsible for parsing, validating, formatting, and packaging rules. It also integrates with configuration files in the “etc” folder (including ECS and Beats schemas) and rule locking/versioning.
• Rule Storage: The “rules/” directory holds the production detection rules in .toml format. In addition, building block rules are in “rules_building_block/”. These are the artifacts that ultimately get imported into Elastic Security.
• Threat Hunting Component: The “hunting/” directory contains an additional module for threat hunting queries and associated scripts. It acts as a complementary component to the detection rules, and it is independently testable.
• Integration Libraries: The project includes two Python libraries in “lib/kibana” and “lib/kql”. The kibana library encapsulates API calls and functions to interface with Kibana’s Detection Engine, while the kql library handles the parsing, transformation, and validation of Kibana Query Language expressions.
• Automated Testing & CI/CD Workflows: The “tests/” directory contains unit tests and integration tests for the rules and helper utilities. Additionally, the .github folder with its workflows and issue/PR templates integrates automated code quality and release management.
• Dependency & Environment Management: The Makefile and pip configuration (pyproject.toml, pre-commit config, etc.) specify toolchain dependencies. Additionally, instructions are provided for installing this project into a Python 3.12+ virtual environment.
Relationships and Interactions
• User Interaction Flow: Developers interact with the repository via the CLI (and possibly via integrated IDE tools) to create and update rule files. Their changes are version controlled and then pushed to GitHub.
• CI/CD Pipeline: Upon commit, GitHub Actions (shown by numerous workflow files inside .github/workflows) kick off unit tests and validation commands (using “test”, “validate-rule” and “toml-lint”) that ensure rule correctness.
• Integration Points: The detection_rules core module communicates with the Kibana integration library to push or synchronize packaged rules with Elastic Security’s Detection Engine. Simultaneously, the kql library is used to convert or validate query language expressions.
• Testing & Validation: The “tests/” module runs unit and integration tests against each component (CLI, rule parser, packaging, transformation utilities, and even remote rule validation). This ensures high-quality output before deployment.
• External Dependencies: The system depends on external Python libraries (such as requests, Click, jsonschema) and on external services like Kibana and Elasticsearch. These can be shown as external systems that interact with the core modules during rule ingestion or update.
Architectural Patterns & Design Principles
• Modular Design: Each piece (CLI, core rule handling, hunting queries, external API integrations, and testing routines) is separated into its own directory and Python package to maintain clean separation of concerns.
• Extensibility & Reusability: The libraries (kibana and kql) are built as reusable Python packages. Similarly, rule definitions are stored in a declarative format (.toml), facilitating easy updates and reuse.
• Automation & Integration: The repository leverages GitHub workflows for CI/CD (automated testing, linting, release management) and uses a Makefile to standardize environment setup.
• “Detections as Code” Paradigm: The design supports turning rule creation into a codified process, making it clear how each part—from rule creation, validation, packaging, to integration—is handled.
– Start with end-user or developer initiating a CLI command.
– Break the system into layers:
• Input Layer: CLI and rule creation interfaces.
• Processing Layer: Core detection_rules module along with libraries (kibana and kql) and configuration (etc).
• Storage Layer: File system folders hosting rule definitions.
• Testing & Automation Layer: CI/CD pipelines (GitHub workflows), unit tests in the tests/ directory.
• External Integration: Kibana and Elastic Security Detection Engine as the endpoint for rule deployment.