How phish-report/iok works
Project Type and Purpose
• IOK is an open-source command-line detection tool & library written in Go.
• It implements a Sigma-style “indicator” language for detecting phishing-site techniques, kits, and threat-actors.
• It ships with a large set of YAML rule files under indicators/ and supports multiple input sources (live HTTP, URLScan.io, etc.).
High-Level Architecture
• Components:
• CLI Frontend
• Input Adapters
– nethttp (live HTTP fetch via Go’s net/http)
– urlscanio (fetch and parse URLScan.io API responses)
• Core Rule Engine
– Loads YAML rules from indicators/
– Parses each rule’s metadata & Sigma conditions
– Executes pattern matching against the page fields (title, html, dom, js, css, headers, cookies, requests)
• Output & Reporting
– Prints matches to stdout or integrates with other tools
• Auxiliary Tools
– tools/internal/verify-rule-references (CI rule-reference checker)
– tools/urlscan-iok (wrapper to fetch URLScan.io and hand off to the core)
Relationships and Data Flow
• User invokes the CLI (e.g. iok scan https://phish.site).
• CLI selects an Input Adapter:
• For live sites, nethttp.go fetches HTML, headers, cookies, embedded assets.
• For URLScan.io, urlscanio.go retrieves a JSON snapshot.
• Adapter normalizes data into a common schema (title, html, dom, js, css, cookies, headers, requests).
• Core Engine:
• Reads every .yml in indicators/ using util.go.
• Evaluates each rule’s Sigma conditions against the schema.
• Emits a match if all conditions pass.
• Output is streamed back to the CLI, which formats it for the user.
• CI pipelines (.github/workflows/*.yml) run unit tests (*_test.go), snapshot tests, and the rule-reference verifier.
Key Architectural Patterns & Principles
• Modular Adapters: each data source (HTTP, URLScan) is a pluggable module implementing a common interface.
• Separation of Concerns:
• Fetch & normalize vs. rule evaluation vs. output formatting.
• Rule definition (YAML) is decoupled from execution logic (Go code).
• Extensibility: new adapters or output formats can be added without touching core matching logic.
• Test-Driven: snapshot tests under .snapshots/, unit tests in _test.go files, plus CI-driven rule verification.
Technologies & Frameworks
• Go (modules: go.mod, go.sum)
• net/http (live HTTP adapter)
• YAML parsing (rules loader)
• URLScan.io API client
• Sigma detection paradigm (conditions on page fields)
• GitHub Actions for CI