# projectdiscovery/subfinder architecture

> Architecture diagram of the GitHub repository projectdiscovery/subfinder, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/projectdiscovery/subfinder
- Repository: https://github.com/projectdiscovery/subfinder

## Overview

1. **What this repo is**
   - `subfinder` is a Go CLI/library for **passive subdomain enumeration**.
   - Its core architecture is a pipeline: **input domains → passive source collection → normalization/deduping → optional DNS validation → output**.

2. **Process entrypoint**
   - `cmd/subfinder/main.go` is the executable entrypoint.
   - It bootstraps the CLI flags, config loading, and hands control to the runner layer.
   - `examples/main.go` shows the library-style embedding/use case.

3. **Orchestration layer**
   - `pkg/runner` is the central coordinator.
   - It owns flag/config interpretation, validation, initialization, enumeration, stats, and output formatting.
   - Key responsibilities are split across `initialize.go`, `validate.go`, `enumerate.go`, `outputter.go`, `stats.go`, and `options.go`.

4. **Input and execution modes**
   - The tool accepts either a single domain list (`-d`) or a file of domains (`-dL`).
   - It supports source selection/exclusion, recursive-only sources, source filtering, and collection of source metadata in JSON output.
   - A library mode exists, but the architecture is still centered on the same runner pipeline.

5. **Passive source subsystem**
   - `pkg/passive` and `pkg/subscraping` are the data-collection engine.
   - `pkg/subscraping` provides the scraping/extraction framework plus common HTTP-agent and parsing utilities.
   - Individual provider adapters live under `pkg/subscraping/sources/*`, one package per external source.

6. **External data providers**
   - The source packages query many external services such as `crtsh`, `censys`, `github`, `shodan`, `securitytrails`, `virustotal`, `waybackarchive`, `urlscan`, `fofa`, and others.
   - Each source adapter encapsulates its own request/response shape, auth requirements, and parsing logic.
   - API keys and provider-specific settings are injected via provider config rather than hardcoded.

7. **Rate limiting and source control**
   - Passive collection has built-in throttling to respect provider limits.
   - The README’s `-rl` and `-rls` flags map to global and per-provider request limits.
   - Some sources are auth-enabled, and the repo includes tests for both authenticated and unauthenticated source behavior.

8. **Normalization, filtering, and aggregation**
   - Collected records are normalized into a common subdomain representation before they leave the collector layer.
   - The runner applies match/filter rules, deduplication, and source attribution.
   - Recursive-capable sources are treated differently from sources that only answer at the apex domain level.

9. **Active verification path**
   - `pkg/resolve` is the DNS verification layer.
   - It performs lookups against configured resolvers, supports concurrency, and eliminates wildcard noise.
   - If active mode is enabled, validated subdomains can also emit IPs.

10. **Output and integration surfaces**
   - Output is handled by `pkg/runner/outputter.go` and can go to stdout, files, JSONL, or an output directory for list mode.
   - The project also supports proxying, custom resolver lists, and update/version behaviors through the runner/config layer.
   - Environment-variable overrides (`SUBFINDER_CONFIG`, `SUBFINDER_PROVIDER_CONFIG`) affect config resolution at startup.

11. **Configuration boundary**
   - Runtime behavior is driven by `config.yaml` and `provider-config.yaml` style inputs, with defaults under the user config directory.
   - This cleanly separates the binary’s orchestration from provider credentials and operational tuning.

12. **Testing and delivery**
   - Tests are concentrated around source adapters, rate limiting, and enumeration behavior.
   - CI/release tooling in `.github/workflows`, plus `goreleaser` and Docker support, indicate this is shipped as a standalone binary and container image, not as a long-running service.

Main entry points: [`pkg/runner/runner.go`](https://github.com/projectdiscovery/subfinder/blob/dev/pkg/runner/runner.go) (Runner), [`pkg/passive/sources.go`](https://github.com/projectdiscovery/subfinder/blob/dev/pkg/passive/sources.go) (Providers), [`pkg/subscraping/extractor.go`](https://github.com/projectdiscovery/subfinder/blob/dev/pkg/subscraping/extractor.go) (Subscraping), [`pkg/runner/config.go`](https://github.com/projectdiscovery/subfinder/blob/dev/pkg/runner/config.go) (Config)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_cli["CLI entrypoints"]
  node_cmd_main["subfinder<br/>cli entrypoint<br/>[main.go]"]
  node_examples_main["Example use<br/>library example<br/>[main.go]"]
end

subgraph group_core["Orchestration"]
  node_runner["Runner<br/>pipeline coordinator<br/>[runner.go]"]
  node_options["Options<br/>flag model<br/>[options.go]"]
  node_config["Config<br/>[config.go]"]
  node_initialize["Init<br/>startup setup<br/>[initialize.go]"]
  node_enumerate["Enumerate<br/>execution stage<br/>[enumerate.go]"]
  node_outputter["Output<br/>sink formatter<br/>[outputter.go]"]
  node_stats["Stats<br/>telemetry<br/>[stats.go]"]
end

subgraph group_collect["Passive collection"]
  node_passive["Passive<br/>collector facade<br/>[passive.go]"]
  node_subscraping["Subscraping<br/>scraping framework<br/>[extractor.go]"]
  node_sources_root["Providers<br/>source registry<br/>[sources.go]"]
  node_github_source["GitHub<br/>provider adapter<br/>[github.go]"]
  node_crtsh_source["crt.sh<br/>provider adapter<br/>[crtsh.go]"]
  node_shodan_source["Shodan<br/>provider adapter<br/>[shodan.go]"]
end

subgraph group_verify["DNS validation"]
  node_validate["Validate<br/>verification gate<br/>[validate.go]"]
  node_resolve["Resolve<br/>dns verifier<br/>[resolve.go]"]
  node_client["Resolvers<br/>dns client<br/>[client.go]"]
end

subgraph group_infra["Delivery"]
  node_release["Release<br/>delivery tooling<br/>[.goreleaser.yml]"]
end

node_cmd_main -->|"starts"| node_runner
node_examples_main -->|"embeds"| node_runner
node_runner -->|"reads"| node_options
node_runner -->|"loads"| node_config
node_runner -->|"initializes"| node_initialize
node_runner -->|"drives"| node_enumerate
node_runner -->|"optionally verifies"| node_validate
node_runner -->|"formats"| node_outputter
node_runner -->|"tracks"| node_stats
node_enumerate -->|"collects via"| node_passive
node_passive -->|"uses"| node_subscraping
node_passive -->|"selects"| node_sources_root
node_sources_root -.->|"includes"| node_github_source
node_sources_root -.->|"includes"| node_crtsh_source
node_sources_root -.->|"includes"| node_shodan_source
node_subscraping -->|"adapts"| node_github_source
node_subscraping -->|"adapts"| node_crtsh_source
node_subscraping -->|"adapts"| node_shodan_source
node_validate -->|"uses"| node_resolve
node_resolve -->|"configures"| node_client
node_config -->|"supplies"| node_resolve
node_config -->|"supplies"| node_sources_root
node_release -.->|"packages"| node_cmd_main

click node_cmd_main "https://github.com/projectdiscovery/subfinder/blob/dev/cmd/subfinder/main.go"
click node_examples_main "https://github.com/projectdiscovery/subfinder/blob/dev/examples/main.go"
click node_runner "https://github.com/projectdiscovery/subfinder/blob/dev/pkg/runner/runner.go"
click node_options "https://github.com/projectdiscovery/subfinder/blob/dev/pkg/runner/options.go"
click node_config "https://github.com/projectdiscovery/subfinder/blob/dev/pkg/runner/config.go"
click node_initialize "https://github.com/projectdiscovery/subfinder/blob/dev/pkg/runner/initialize.go"
click node_enumerate "https://github.com/projectdiscovery/subfinder/blob/dev/pkg/runner/enumerate.go"
click node_validate "https://github.com/projectdiscovery/subfinder/blob/dev/pkg/runner/validate.go"
click node_outputter "https://github.com/projectdiscovery/subfinder/blob/dev/pkg/runner/outputter.go"
click node_stats "https://github.com/projectdiscovery/subfinder/blob/dev/pkg/runner/stats.go"
click node_passive "https://github.com/projectdiscovery/subfinder/blob/dev/pkg/passive/passive.go"
click node_subscraping "https://github.com/projectdiscovery/subfinder/blob/dev/pkg/subscraping/extractor.go"
click node_sources_root "https://github.com/projectdiscovery/subfinder/blob/dev/pkg/passive/sources.go"
click node_github_source "https://github.com/projectdiscovery/subfinder/blob/dev/pkg/subscraping/sources/github/github.go"
click node_crtsh_source "https://github.com/projectdiscovery/subfinder/blob/dev/pkg/subscraping/sources/crtsh/crtsh.go"
click node_shodan_source "https://github.com/projectdiscovery/subfinder/blob/dev/pkg/subscraping/sources/shodan/shodan.go"
click node_resolve "https://github.com/projectdiscovery/subfinder/blob/dev/pkg/resolve/resolve.go"
click node_client "https://github.com/projectdiscovery/subfinder/blob/dev/pkg/resolve/client.go"
click node_release "https://github.com/projectdiscovery/subfinder/blob/dev/.goreleaser.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cmd_main,node_examples_main toneBlue
class node_runner,node_options,node_config,node_initialize,node_enumerate,node_outputter,node_stats toneAmber
class node_passive,node_subscraping,node_sources_root,node_github_source,node_crtsh_source,node_shodan_source toneMint
class node_validate,node_resolve,node_client toneRose
class node_release toneIndigo
```

## Components

### CLI entrypoints

Binary and library entry surfaces

- [subfinder](https://github.com/projectdiscovery/subfinder/blob/dev/cmd/subfinder/main.go): cli entrypoint · Executable bootstrap for flags, config, and runner startup · `cmd/subfinder/main.go`
- [Example use](https://github.com/projectdiscovery/subfinder/blob/dev/examples/main.go): library example · Library-style embedding of the runner · `examples/main.go`

### Orchestration

Runner, config, and pipeline control

- [Runner](https://github.com/projectdiscovery/subfinder/blob/dev/pkg/runner/runner.go): pipeline coordinator · Central orchestration of enumeration, validation, stats, and output · `pkg/runner/runner.go`
- [Options](https://github.com/projectdiscovery/subfinder/blob/dev/pkg/runner/options.go): flag model · CLI and runtime option parsing and normalization · `pkg/runner/options.go`
- [Config](https://github.com/projectdiscovery/subfinder/blob/dev/pkg/runner/config.go): runtime config · Loads user and provider configuration with environment overrides · `pkg/runner/config.go`
- [Init](https://github.com/projectdiscovery/subfinder/blob/dev/pkg/runner/initialize.go): startup setup · Prepares sources, resolvers, and execution state · `pkg/runner/initialize.go`
- [Enumerate](https://github.com/projectdiscovery/subfinder/blob/dev/pkg/runner/enumerate.go): execution stage · Drives domain input through passive collection and aggregation · `pkg/runner/enumerate.go`
- [Output](https://github.com/projectdiscovery/subfinder/blob/dev/pkg/runner/outputter.go): sink formatter · Formats results to stdout, files, JSONL, or list-mode outputs · `pkg/runner/outputter.go`
- [Stats](https://github.com/projectdiscovery/subfinder/blob/dev/pkg/runner/stats.go): telemetry · Tracks enumeration progress and execution summary · `pkg/runner/stats.go`

### Passive collection

Source scraping and aggregation

- [Passive](https://github.com/projectdiscovery/subfinder/blob/dev/pkg/passive/passive.go): collector facade · Top-level passive enumeration package · `pkg/passive/passive.go`
- [Subscraping](https://github.com/projectdiscovery/subfinder/blob/dev/pkg/subscraping/extractor.go): scraping framework · Common extraction framework, HTTP agent, and parsing utilities · `pkg/subscraping/extractor.go`
- [Providers](https://github.com/projectdiscovery/subfinder/blob/dev/pkg/passive/sources.go): source registry · Registry and grouping of external source adapters · `pkg/passive/sources.go`
- [GitHub](https://github.com/projectdiscovery/subfinder/blob/dev/pkg/subscraping/sources/github/github.go): provider adapter · Representative authenticated source adapter with token management · `pkg/subscraping/sources/github/github.go`
- [crt.sh](https://github.com/projectdiscovery/subfinder/blob/dev/pkg/subscraping/sources/crtsh/crtsh.go): provider adapter · Representative passive certificate transparency source · `pkg/subscraping/sources/crtsh/crtsh.go`
- [Shodan](https://github.com/projectdiscovery/subfinder/blob/dev/pkg/subscraping/sources/shodan/shodan.go): provider adapter · Representative API-backed source adapter · `pkg/subscraping/sources/shodan/shodan.go`

### DNS validation

Active resolution and wildcard filtering

- [Validate](https://github.com/projectdiscovery/subfinder/blob/dev/pkg/runner/validate.go): verification gate · Coordinates optional DNS validation before output · `pkg/runner/validate.go`
- [Resolve](https://github.com/projectdiscovery/subfinder/blob/dev/pkg/resolve/resolve.go): dns verifier · DNS lookup engine with concurrency and wildcard suppression · `pkg/resolve/resolve.go`
- [Resolvers](https://github.com/projectdiscovery/subfinder/blob/dev/pkg/resolve/client.go): dns client · Resolver client and lookup configuration · `pkg/resolve/client.go`

### Delivery

Build, release, and packaging

- [Release](https://github.com/projectdiscovery/subfinder/blob/dev/.goreleaser.yml): delivery tooling · Build and packaging automation for binaries and containers · `.goreleaser.yml`

## Connections

- subfinder → Runner: starts · CLI hands control to the orchestration layer
- Example use → Runner: embeds · Example shows library-style reuse
- Runner → Options: reads · Runner interprets flags and execution mode
- Runner → Config: loads · Runtime configuration is resolved at startup
- Runner → Init: initializes · Prepares execution state and dependencies
- Runner → Enumerate: drives · Main enumeration pipeline
- Runner → Validate: optionally verifies · Active DNS validation is gated by options
- Runner → Output: formats · Results are emitted through the output layer
- Runner → Stats: tracks · Execution progress and summary are recorded
- Enumerate → Passive: collects via · Enumeration delegates to passive collection
- Passive → Subscraping: uses · Collector relies on shared scraping and extraction utilities
- Passive → Providers: selects · Source registry controls enabled providers
- Providers → GitHub: includes · Representative provider adapter
- Providers → crt.sh: includes · Representative provider adapter
- Providers → Shodan: includes · Representative provider adapter
- Subscraping → GitHub: adapts · Source adapters implement provider-specific requests and parsing
- Subscraping → crt.sh: adapts · Source adapters implement provider-specific requests and parsing
- Subscraping → Shodan: adapts · Source adapters implement provider-specific requests and parsing
- Validate → Resolve: uses · DNS validation resolves discovered subdomains
- Resolve → Resolvers: configures · Resolver client provides lookup behavior
- Config → Resolve: supplies · Resolver settings and custom lists flow from config
- Config → Providers: supplies · Provider credentials and limits are injected from provider config
- Release → subfinder: packages · Binary release and container build artifacts
