# mongodb/kingfisher architecture

> Architecture diagram of the GitHub repository mongodb/kingfisher, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mongodb/kingfisher
- Repository: https://github.com/mongodb/kingfisher

## Overview

##### Project Type and Purpose
- Kingfisher is a Rust-based, console/CLI tool for secret scanning and live validation.
- It is not a web or full-stack application; it is an extensible, high-performance command-line utility.
- Key capabilities:
- Scan multiple targets (local repos, GitHub, GitLab, S3, Docker images, Jira issues, Slack messages)
- Use Hyperscan for regex matching and Tree-Sitter for language-aware parsing
- Ship hundreds of built-in rules (YAML) and load custom rule files at runtime
- Perform live validation via cloud-provider and service APIs
- Output results in various formats (JSON, SARIF, pretty, BSON)
- Baseline management to suppress known findings

##### File-Structure Analysis
- Top-level crates and config:
- Cargo.toml, src/, data/, docs/, Dockerfile, Makefile, testdata/, tests/
- data/default: sample scripts, ignore rules, cleanup hooks
- data/rules: YAML rule definitions for hundreds of services
- Source code modules in src/:
- cli/: command-definition and argument parsing
- scanner/: orchestrates concurrent scanning, archive extraction, repository enumeration
- parser/: language-aware parsing hooks (Tree-Sitter queries)
- matcher.rs: runs Hyperscan regexes against file contents
- rule_loader.rs: loads built-in and custom YAML rules, compiles them into matchers
- validation/: live validation logic per service (AWS, Azure, GCP, HTTP, JWT, etc.)
- reporter/: multiple output formats (JSON, SARIF, pretty, BSON)
- baseline.rs: baseline management
- util/*: helpers (entropy, decompression, content type, etc.)
- Tests and benchmarks:
- tests/: integration and smoke tests covering CLI and scanning targets
- docs/benchmark: standalone Go benchmark harness for comparison

##### Key Components
a. CLI Frontend
- Entry point (main.rs → binary.rs)
- Global command dispatcher (cli/global.rs)
- Subcommands (scan, rules list/check, github repos, etc.)
b. Configuration & Rule Loader
- data/default + data/rules files
- rule_loader.rs compiles YAML definitions into Hyperscan patterns and validation logic
c. Target Enumerators & Fetchers
- filesystem scanner (scan local dirs, git metadata)
- git repos (local, git-url, GitHub, GitLab) via git_repo_enumerator.rs
- S3 bucket scanner (s3.rs)
- Docker image scanner (scanner/docker.rs)
- Jira (jira.rs), Slack (slack.rs) fetchers
d. Scanner Pool & Workers
- scanner_pool.rs: thread pool managing scan tasks
- scanner/runner.rs: worker logic
- enumeration → extraction → detection
e. Content Processing Pipeline
- decompress.rs for archives
- parser module: Tree-Sitter queries for language syntax awareness
- matcher.rs: Hyperscan engine scanning raw text or token streams
f. Findings Storage & Deduplication
- findings_store.rs: collects raw matches
- fingerprint formula (docs/FINGERPRINT.md) → dedupe
g. Validation Subsystem
- validation/ modules call live APIs per service to confirm secret validity
h. Reporting & Output
- reporter/*: JSON, SARIF, pretty, BSON
- baseline.rs: load or update baseline file to suppress known secrets
i. Update Checker
- binary self-update logic via GitHub API

##### Relationships & Data Flow
- User invokes CLI
→ CLI parser determines subcommand
→ Rule Loader loads built-ins and user rules
→ Configuration object passed to scan runner
→ Target Enumerator fetches files or objects
→ Scanner Pool enqueues content chunks
→ Workers decompress archives and feed content to parser & matcher
→ Matcher yields findings → Findings Store → Deduplication
→ For each finding: call Validation subsystem if enabled
→ Reporter formats results → writes to STDOUT or file
→ Optionally update baseline file
→ Exit code reflects findings/validation status

##### Architectural Patterns & Design Principles
- Modular separation: CLI, scanning, parsing, matching, validation, reporting
- Pipeline pattern: enumeration → processing → detection → output
- Thread-pool concurrency for throughput
- Plugin-style rule loader allowing runtime extension (built-ins + custom)
- Single binary, no external services except for live validation calls

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CLI Frontend
    subgraph "CLI Frontend"
        direction TB
        Main["main.rs"]:::core
        Binary["binary.rs"]:::core
        CliMod["cli/mod.rs"]:::core
        Global["cli/global.rs"]:::core
        ScanCmd["cli/commands/scan.rs"]:::core
        RulesCmd["cli/commands/rules.rs"]:::core
        GHCmd["cli/commands/github.rs"]:::core
        GLCmd["cli/commands/gitlab.rs"]:::core
        OutCmd["cli/commands/output.rs"]:::core
    end

    %% Configuration & Rule Loader
    subgraph "Configuration & Rule Loader"
        direction TB
        DataDefault["data/default/"]:::config
        DataRules["data/rules/"]:::config
        RuleLoader["rule_loader.rs"]:::config
        RulesRs["rules.rs"]:::config
        RulesLib["rules/lib.rs"]:::config
        RulesRule["rules/rule.rs"]:::config
        RulesUtil["rules/util.rs"]:::config
        RulesDB["rules_database.rs"]:::config
    end

    %% Target Enumerators & Fetchers
    subgraph "Fetchers" 
        direction TB
        GitEnum["git_repo_enumerator.rs"]:::external
        ReposEnum["scanner/repos.rs"]:::external
        Enumerate["scanner/enumerate.rs"]:::external
        DockerEnum["scanner/docker.rs"]:::external
        S3Enum["s3.rs"]:::external
        JiraEnum["jira.rs"]:::external
        SlackEnum["slack.rs"]:::external
    end

    %% Scanner Pool & Workers
    subgraph "Scanner Pool & Workers"
        direction TB
        ScannerPool["scanner_pool.rs"]:::core
        Runner["scanner/runner.rs"]:::core
        Processing["scanner/processing.rs"]:::core
        Summary["scanner/summary.rs"]:::core
        ScannerUtil["scanner/util.rs"]:::core
    end

    %% Content Processing Pipeline
    subgraph "Content Processing Pipeline"
        direction TB
        Decompress["decompress.rs"]:::core
        Parser["parser.rs"]:::core
        Queries["parser/queries.rs"]:::core
        Matcher["matcher.rs"]:::core
    end

    %% Findings & Deduplication
    subgraph "Findings Storage & Deduplication"
        direction TB
        FindingsStore["findings_store.rs"]:::core
        Fingerprint["FINGERPRINT.md"]:::core
    end

    %% Validation Subsystem
    subgraph "Validation Subsystem"
        direction TB
        AWSVal["validation/aws.rs"]:::external
        AzureVal["validation/azure.rs"]:::external
        GCPVal["validation/gcp.rs"]:::external
        HTTPVal["validation/httpvalidation.rs"]:::external
        JWTVal["validation/jwt.rs"]:::external
        ValUtil["validation/utils.rs"]:::external
        ValidationMain["validation.rs"]:::external
    end

    %% Reporting & Output
    subgraph "Reporting & Output"
        direction TB
        JSONFmt["reporter/json_format.rs"]:::core
        SARIFFmt["reporter/sarif_format.rs"]:::core
        PrettyFmt["reporter/pretty_format.rs"]:::core
        BSONFmt["reporter/bson_format.rs"]:::core
        Styles["reporter/styles.rs"]:::core
        ReporterMain["reporter.rs"]:::core
        Baseline["baseline.rs"]:::core
    end

    %% Update Checker
    Update["update.rs"]:::core

    %% Data Flow
    Main -->|invokes| Global
    Global -->|dispatches| ScanCmd & RulesCmd & GHCmd & GLCmd & OutCmd
    ScanCmd -->|uses| RuleLoader
    RulesCmd -->|uses| RuleLoader
    RuleLoader --> DataDefault & DataRules
    RuleLoader --> RulesRs & RulesLib & RulesRule & RulesUtil & RulesDB
    ScanCmd --> ScannerPool
    ScannerPool --> Runner
    Runner --> GitEnum & ReposEnum & Enumerate & DockerEnum & S3Enum & JiraEnum & SlackEnum
    Runner --> Decompress
    Decompress --> Parser
    Parser --> Queries
    Queries --> Matcher
    Matcher --> FindingsStore
    FindingsStore --> Fingerprint
    FindingsStore -->|for each| ValidationMain
    ValidationMain --> AWSVal & AzureVal & GCPVal & HTTPVal & JWTVal & ValUtil
    ValidationMain --> ReporterMain
    ReporterMain --> JSONFmt & SARIFFmt & PrettyFmt & BSONFmt & Styles & Baseline
    Global --> Update

    %% Click Events
    click Main "https://github.com/mongodb/kingfisher/blob/main/src/main.rs"
    click Binary "https://github.com/mongodb/kingfisher/blob/main/src/binary.rs"
    click CliMod "https://github.com/mongodb/kingfisher/blob/main/src/cli/mod.rs"
    click Global "https://github.com/mongodb/kingfisher/blob/main/src/cli/global.rs"
    click ScanCmd "https://github.com/mongodb/kingfisher/blob/main/src/cli/commands/scan.rs"
    click RulesCmd "https://github.com/mongodb/kingfisher/blob/main/src/cli/commands/rules.rs"
    click GHCmd "https://github.com/mongodb/kingfisher/blob/main/src/cli/commands/github.rs"
    click GLCmd "https://github.com/mongodb/kingfisher/blob/main/src/cli/commands/gitlab.rs"
    click OutCmd "https://github.com/mongodb/kingfisher/blob/main/src/cli/commands/output.rs"
    click DataDefault "https://github.com/mongodb/kingfisher/tree/main/data/default/"
    click DataRules "https://github.com/mongodb/kingfisher/tree/main/data/rules/"
    click RuleLoader "https://github.com/mongodb/kingfisher/blob/main/src/rule_loader.rs"
    click RulesRs "https://github.com/mongodb/kingfisher/blob/main/src/rules.rs"
    click RulesLib "https://github.com/mongodb/kingfisher/blob/main/src/rules/lib.rs"
    click RulesRule "https://github.com/mongodb/kingfisher/blob/main/src/rules/rule.rs"
    click RulesUtil "https://github.com/mongodb/kingfisher/blob/main/src/rules/util.rs"
    click RulesDB "https://github.com/mongodb/kingfisher/blob/main/src/rules_database.rs"
    click GitEnum "https://github.com/mongodb/kingfisher/blob/main/src/git_repo_enumerator.rs"
    click ReposEnum "https://github.com/mongodb/kingfisher/blob/main/src/scanner/repos.rs"
    click Enumerate "https://github.com/mongodb/kingfisher/blob/main/src/scanner/enumerate.rs"
    click DockerEnum "https://github.com/mongodb/kingfisher/blob/main/src/scanner/docker.rs"
    click S3Enum "https://github.com/mongodb/kingfisher/blob/main/src/s3.rs"
    click JiraEnum "https://github.com/mongodb/kingfisher/blob/main/src/jira.rs"
    click SlackEnum "https://github.com/mongodb/kingfisher/blob/main/src/slack.rs"
    click ScannerPool "https://github.com/mongodb/kingfisher/blob/main/src/scanner_pool.rs"
    click Runner "https://github.com/mongodb/kingfisher/blob/main/src/scanner/runner.rs"
    click Processing "https://github.com/mongodb/kingfisher/blob/main/src/scanner/processing.rs"
    click Summary "https://github.com/mongodb/kingfisher/blob/main/src/scanner/summary.rs"
    click ScannerUtil "https://github.com/mongodb/kingfisher/blob/main/src/scanner/util.rs"
    click Decompress "https://github.com/mongodb/kingfisher/blob/main/src/decompress.rs"
    click Parser "https://github.com/mongodb/kingfisher/blob/main/src/parser.rs"
    click Queries "https://github.com/mongodb/kingfisher/blob/main/src/parser/queries.rs"
    click Matcher "https://github.com/mongodb/kingfisher/blob/main/src/matcher.rs"
    click FindingsStore "https://github.com/mongodb/kingfisher/blob/main/src/findings_store.rs"
    click Fingerprint "https://github.com/mongodb/kingfisher/blob/main/docs/FINGERPRINT.md"
    click AWSVal "https://github.com/mongodb/kingfisher/blob/main/src/validation/aws.rs"
    click AzureVal "https://github.com/mongodb/kingfisher/blob/main/src/validation/azure.rs"
    click GCPVal "https://github.com/mongodb/kingfisher/blob/main/src/validation/gcp.rs"
    click HTTPVal "https://github.com/mongodb/kingfisher/blob/main/src/validation/httpvalidation.rs"
    click JWTVal "https://github.com/mongodb/kingfisher/blob/main/src/validation/jwt.rs"
    click ValUtil "https://github.com/mongodb/kingfisher/blob/main/src/validation/utils.rs"
    click ValidationMain "https://github.com/mongodb/kingfisher/blob/main/src/validation.rs"
    click JSONFmt "https://github.com/mongodb/kingfisher/blob/main/src/reporter/json_format.rs"
    click SARIFFmt "https://github.com/mongodb/kingfisher/blob/main/src/reporter/sarif_format.rs"
    click PrettyFmt "https://github.com/mongodb/kingfisher/blob/main/src/reporter/pretty_format.rs"
    click BSONFmt "https://github.com/mongodb/kingfisher/blob/main/src/reporter/bson_format.rs"
    click Styles "https://github.com/mongodb/kingfisher/blob/main/src/reporter/styles.rs"
    click ReporterMain "https://github.com/mongodb/kingfisher/blob/main/src/reporter.rs"
    click Baseline "https://github.com/mongodb/kingfisher/blob/main/src/baseline.rs"
    click Update "https://github.com/mongodb/kingfisher/blob/main/src/update.rs"

    %% Styles
    classDef core fill:#CFE2F3,stroke:#0B5394,color:#073763
    classDef external fill:#D9EAD3,stroke:#6AA84F,color:#274E13
    classDef config fill:#FCE5CD,stroke:#E69138,color:#974F0C
```
