How mongodb/kingfisher works
Project Type and Purpose
• Kingfisher is a Rust-based, console/CLI tool for secret scanning and live validation.
• It is not a web or full-stack application; it is an extensible, high-performance command-line utility.
• Key capabilities:
• Scan multiple targets (local repos, GitHub, GitLab, S3, Docker images, Jira issues, Slack messages)
• Use Hyperscan for regex matching and Tree-Sitter for language-aware parsing
• Ship hundreds of built-in rules (YAML) and load custom rule files at runtime
• Perform live validation via cloud-provider and service APIs
• Output results in various formats (JSON, SARIF, pretty, BSON)
• Baseline management to suppress known findings
File-Structure Analysis
• Top-level crates and config:
• Cargo.toml, src/, data/, docs/, Dockerfile, Makefile, testdata/, tests/
• data/default: sample scripts, ignore rules, cleanup hooks
• data/rules: YAML rule definitions for hundreds of services
• Source code modules in src/:
• cli/: command-definition and argument parsing
• scanner/: orchestrates concurrent scanning, archive extraction, repository enumeration
• parser/: language-aware parsing hooks (Tree-Sitter queries)
• matcher.rs: runs Hyperscan regexes against file contents
• rule_loader.rs: loads built-in and custom YAML rules, compiles them into matchers
• validation/: live validation logic per service (AWS, Azure, GCP, HTTP, JWT, etc.)
• reporter/: multiple output formats (JSON, SARIF, pretty, BSON)
• baseline.rs: baseline management
• util/*: helpers (entropy, decompression, content type, etc.)
• Tests and benchmarks:
• tests/: integration and smoke tests covering CLI and scanning targets
• docs/benchmark: standalone Go benchmark harness for comparison
Key Components
a. CLI Frontend
• Entry point (main.rs → binary.rs)
• Global command dispatcher (cli/global.rs)
• Subcommands (scan, rules list/check, github repos, etc.)
b. Configuration & Rule Loader
• data/default + data/rules files
• rule_loader.rs compiles YAML definitions into Hyperscan patterns and validation logic
c. Target Enumerators & Fetchers
• filesystem scanner (scan local dirs, git metadata)
• git repos (local, git-url, GitHub, GitLab) via git_repo_enumerator.rs
• S3 bucket scanner (s3.rs)
• Docker image scanner (scanner/docker.rs)
• Jira (jira.rs), Slack (slack.rs) fetchers
d. Scanner Pool & Workers
• scanner_pool.rs: thread pool managing scan tasks
• scanner/runner.rs: worker logic
• enumeration → extraction → detection
e. Content Processing Pipeline
• decompress.rs for archives
• parser module: Tree-Sitter queries for language syntax awareness
• matcher.rs: Hyperscan engine scanning raw text or token streams
f. Findings Storage & Deduplication
• findings_store.rs: collects raw matches
• fingerprint formula (docs/FINGERPRINT.md) → dedupe
g. Validation Subsystem
• validation/ modules call live APIs per service to confirm secret validity
h. Reporting & Output
• reporter/*: JSON, SARIF, pretty, BSON
• baseline.rs: load or update baseline file to suppress known secrets
i. Update Checker
• binary self-update logic via GitHub API
Relationships & Data Flow
• User invokes CLI
→ CLI parser determines subcommand
→ Rule Loader loads built-ins and user rules
→ Configuration object passed to scan runner
→ Target Enumerator fetches files or objects
→ Scanner Pool enqueues content chunks
→ Workers decompress archives and feed content to parser & matcher
→ Matcher yields findings → Findings Store → Deduplication
→ For each finding: call Validation subsystem if enabled
→ Reporter formats results → writes to STDOUT or file
→ Optionally update baseline file
→ Exit code reflects findings/validation status
Architectural Patterns & Design Principles
• Modular separation: CLI, scanning, parsing, matching, validation, reporting
• Pipeline pattern: enumeration → processing → detection → output
• Thread-pool concurrency for throughput
• Plugin-style rule loader allowing runtime extension (built-ins + custom)
• Single binary, no external services except for live validation calls