How yamato-security/hayabusa works
Project Type and Purpose
• Hayabusa is a command-line tool designed for rapid forensic analysis and threat hunting via Windows event logs. It reads Windows evtx (or JSON) files, applies detection rules (written in YAML with Sigma compatibility), enriches events (for example with GeoIP data), aggregates timeline data, and finally exports analysis results as CSV, JSON, or HTML. Its design supports both live system analysis and offline batch processing, with multi-threading for improved performance.
Break Down the File Structure into Major Components
• Top-level directories such as “src”, “config”, “rules”, “doc”, and “test_files” indicate clear separation of concerns.
• The “src” directory holds the core Rust code (with submodules like “detections”, “options”, “timeline”, “notify”, etc.) responsible for:
– Parsing logs and handling file I/O.
– Running detection analysis and rule matching.
– Computing and outputting various metrics and timelines.
• The “rules” directory (including a submodule via Git) contains the detection rules (YAML files) and associated configuration (for adjusting rule levels, mapping fields, expand placeholders, etc.).
• Other directories like “doc”, “logs”, and “art” support documentation, sample data, and user art – but the core architecture is in src, rules, and config.
Main Architectural Components
• Command-Line Interface (CLI) & Input Manager:
– Parses arguments (including subcommands like “csv-timeline”, “json-timeline”, “computer-metrics”, “level-tuning”, etc.) and dispatches the workload.
• File Loader & Log Parser:
– Handles various input sources: reading evtx files, JSON log files, live analysis from Windows event log folders.
– Employs the evtx parsing library (or its fork) with support for recovering and carving event records.
• Multi-threaded Processing Core:
– Orchestrates parallel processing of files.
– Ensures each file is fed into the processing engine for performance.
• Detection Engine & Rule Matcher:
– Loads detection rules from the “rules” directory (including Sigma-converted rules).
– Applies rule matching to parsed events, including signature matching and condition filtering.
– Manages special processing features (like “expand list” for dynamic mapping, field data mapping, and level tuning adjustments from the configuration).
• Enrichment & Aggregation Module:
– Optionally integrates with GeoIP data (using supplied MaxMind databases) to add ASN, city, and country details.
– Aggregates metrics (such as computer metrics, event ID metrics, logon summaries, etc.) and prepares timeline data.
• Output & Reporting Module:
– Exports the forensic timeline in multiple formats (CSV, JSON/JSONL, or HTML reports).
– Applies output profiles (minimal, standard, verbose, super-verbose, Timesketch-compatible, etc.) that determine which fields and aliases are included.
• Configuration & Customization Layer:
– Loads global configuration (from “config” and “rules/config” directories) for field mappings, level tuning, critical systems, and abbreviation rules.
– Offers commands to update rules, tune alert levels, and set default output profiles.
• Logging, Error Handling, and Notification System:
– Manages progress indicators (progress bars), logs processing errors, and even triggers notifications when needed.
Relationships and Data Flow
• Start at the CLI layer where the user selects an analysis mode.
• The Input Manager then directs files (or live sources) to the Log Parser.
• Parsed events are dispatched by the multi-threaded core to the Detection Engine.
• The Detection Engine consults loaded YAML detection rules to flag “alerts” versus normal “informational” events.
• Enrichment (such as GeoIP lookup) is then optionally applied.
• Aggregated detection events and metrics are assembled by the Aggregation Module.
• Finally, the Output Module formats the timeline or metrics into the chosen file (CSV, JSON, HTML) using pre-defined output profiles.
• Throughout, configuration files (from “config” and “rules/config”) modify behavior at almost every stage.
Important Architectural Patterns & Technologies
• Modular design: Separate concerns between input, processing, rule matching, enrichment, and output.
• Plugin-like rule processing: The dynamic loading and matching of YAML rules (including Sigma rule conversion) supports high flexibility.
• Multi-threading: Distributed processing of evtx files to improve speed.
• Configurability: With external configuration files for tuning, field mapping, and output formatting.
• Command pattern: The variety of subcommands for metrics, search, timeline, and level tuning each correspond to discrete operations within the tool.