# phish-report/iok architecture

> Architecture diagram of the GitHub repository phish-report/iok, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/phish-report/iok
- Repository: https://github.com/phish-report/iok

## Overview

##### Project Type and Purpose
- IOK is an open-source command-line detection tool & library written in Go.
- It implements a Sigma-style “indicator” language for detecting phishing-site techniques, kits, and threat-actors.
- It ships with a large set of YAML rule files under `indicators/` and supports multiple input sources (live HTTP, URLScan.io, etc.).

##### High-Level Architecture
- Components:
- CLI Frontend
- Input Adapters
– nethttp (live HTTP fetch via Go’s net/http)
– urlscanio (fetch and parse URLScan.io API responses)
- Core Rule Engine
– Loads YAML rules from `indicators/`
– Parses each rule’s metadata & Sigma conditions
– Executes pattern matching against the page fields (title, html, dom, js, css, headers, cookies, requests)
- Output & Reporting
– Prints matches to stdout or integrates with other tools
- Auxiliary Tools
– `tools/internal/verify-rule-references` (CI rule-reference checker)
– `tools/urlscan-iok` (wrapper to fetch URLScan.io and hand off to the core)

##### Relationships and Data Flow
- User invokes the CLI (e.g. `iok scan https://phish.site`).
- CLI selects an Input Adapter:
- For live sites, `nethttp.go` fetches HTML, headers, cookies, embedded assets.
- For URLScan.io, `urlscanio.go` retrieves a JSON snapshot.
- Adapter normalizes data into a common schema (title, html, dom, js, css, cookies, headers, requests).
- Core Engine:
- Reads every `.yml` in `indicators/` using `util.go`.
- Evaluates each rule’s Sigma conditions against the schema.
- Emits a match if all conditions pass.
- Output is streamed back to the CLI, which formats it for the user.
- CI pipelines (`.github/workflows/*.yml`) run unit tests (`*_test.go`), snapshot tests, and the rule-reference verifier.

##### Key Architectural Patterns & Principles
- Modular Adapters: each data source (HTTP, URLScan) is a pluggable module implementing a common interface.
- Separation of Concerns:
- Fetch & normalize vs. rule evaluation vs. output formatting.
- Rule definition (YAML) is decoupled from execution logic (Go code).
- Extensibility: new adapters or output formats can be added without touching core matching logic.
- Test-Driven: snapshot tests under `.snapshots/`, unit tests in `_test.go` files, plus CI-driven rule verification.

##### Technologies & Frameworks
- Go (modules: `go.mod`, `go.sum`)
- net/http (live HTTP adapter)
- YAML parsing (rules loader)
- URLScan.io API client
- Sigma detection paradigm (conditions on page fields)
- GitHub Actions for CI

## Diagram (Mermaid)

```mermaid
flowchart TD
    CLI["CLI Frontend"]:::component
    subgraph "Input Sources"
        Nethttp["nethttp Adapter"]:::component
        URLScanIO["urlscanio Adapter"]:::component
        ExternalURLScan["URLScan.io API"]:::external
    end

    subgraph "Core Rule Engine"
        IndicatorStore["indicators/ (YAML Store)"]:::store
        Util["util.go (Rule Loader)"]:::component
        Engine["Rule Engine"]:::component
    end

    subgraph "Output & Reporting"
        Output["Output Formatter"]:::component
    end

    subgraph "Tools & CI"
        Verifier["verify-rule-references"]:::tool
        URLScanTool["urlscan-iok"]:::tool
        CI_Go["go.yml"]:::tool
        CI_Rules["rules.yml"]:::tool
        Snapshots[".snapshots/"]:::store
        iokTest["iok_test.go"]:::tool
        nethttpTest["nethttp_test.go"]:::tool
        urlscanioTest["urlscanio_test.go"]:::tool
    end

    CLI -->|invokes| Nethttp
    CLI -->|invokes| URLScanIO
    Nethttp -->|fetch HTML| ExternalURLScan
    URLScanIO -->|fetch JSON| ExternalURLScan

    Nethttp -->|"normalized schema"| Engine
    URLScanIO -->|"normalized schema"| Engine
    IndicatorStore -->|loads rules| Util
    Util -->|provides rules| Engine

    Engine -->|"matches"| Output
    Output -->|"stdout/JSON"| CLI

    IndicatorStore -->|used by| Verifier
    CLI -->|tested by| iokTest
    Nethttp -->|tested by| nethttpTest
    URLScanIO -->|tested by| urlscanioTest
    CI_Go -->|runs tests| iokTest
    CI_Go -->|runs tests| nethttpTest
    CI_Go -->|runs tests| urlscanioTest
    CI_Rules -->|runs verifier| Verifier

    click CLI "https://github.com/phish-report/iok/blob/main/iok.go"
    click Nethttp "https://github.com/phish-report/iok/blob/main/nethttp.go"
    click URLScanIO "https://github.com/phish-report/iok/blob/main/urlscanio.go"
    click Util "https://github.com/phish-report/iok/blob/main/util.go"
    click IndicatorStore "https://github.com/phish-report/iok/tree/main/indicators/"
    click Verifier "https://github.com/phish-report/iok/blob/main/tools/internal/verify-rule-references/verify-rule-references.go"
    click URLScanTool "https://github.com/phish-report/iok/blob/main/tools/urlscan-iok/urlscan-iok.go"
    click CI_Go "https://github.com/phish-report/iok/blob/main/.github/workflows/go.yml"
    click CI_Rules "https://github.com/phish-report/iok/blob/main/.github/workflows/rules.yml"
    click Snapshots "https://github.com/phish-report/iok/tree/main/.snapshots/"
    click iokTest "https://github.com/phish-report/iok/blob/main/iok_test.go"
    click nethttpTest "https://github.com/phish-report/iok/blob/main/nethttp_test.go"
    click urlscanioTest "https://github.com/phish-report/iok/blob/main/urlscanio_test.go"

    classDef component fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef store fill:#FFFF99,stroke:#333,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5
    classDef tool fill:#F0F0F0,stroke:#333,stroke-width:1px
```
