How projectdiscovery/subfinder works
1. What this repo is
- subfinder is a Go CLI/library for passive subdomain enumeration.
- Its core architecture is a pipeline: input domains → passive source collection → normalization/deduping → optional DNS validation → output.
2. Process entrypoint
- cmd/subfinder/main.go is the executable entrypoint.
- It bootstraps the CLI flags, config loading, and hands control to the runner layer.
- examples/main.go shows the library-style embedding/use case.
3. Orchestration layer
- pkg/runner is the central coordinator.
- It owns flag/config interpretation, validation, initialization, enumeration, stats, and output formatting.
- Key responsibilities are split across initialize.go, validate.go, enumerate.go, outputter.go, stats.go, and options.go.
4. Input and execution modes
- The tool accepts either a single domain list (-d) or a file of domains (-dL).
- It supports source selection/exclusion, recursive-only sources, source filtering, and collection of source metadata in JSON output.
- A library mode exists, but the architecture is still centered on the same runner pipeline.
5. Passive source subsystem
- pkg/passive and pkg/subscraping are the data-collection engine.
- pkg/subscraping provides the scraping/extraction framework plus common HTTP-agent and parsing utilities.
- Individual provider adapters live under pkg/subscraping/sources/*, one package per external source.
6. External data providers
- The source packages query many external services such as crtsh, censys, github, shodan, securitytrails, virustotal, waybackarchive, urlscan, fofa, and others.
- Each source adapter encapsulates its own request/response shape, auth requirements, and parsing logic.
- API keys and provider-specific settings are injected via provider config rather than hardcoded.
7. Rate limiting and source control
- Passive collection has built-in throttling to respect provider limits.
- The README’s -rl and -rls flags map to global and per-provider request limits.
- Some sources are auth-enabled, and the repo includes tests for both authenticated and unauthenticated source behavior.
8. Normalization, filtering, and aggregation
- Collected records are normalized into a common subdomain representation before they leave the collector layer.
- The runner applies match/filter rules, deduplication, and source attribution.
- Recursive-capable sources are treated differently from sources that only answer at the apex domain level.
9. Active verification path
- pkg/resolve is the DNS verification layer.
- It performs lookups against configured resolvers, supports concurrency, and eliminates wildcard noise.
- If active mode is enabled, validated subdomains can also emit IPs.
10. Output and integration surfaces
- Output is handled by pkg/runner/outputter.go and can go to stdout, files, JSONL, or an output directory for list mode.
- The project also supports proxying, custom resolver lists, and update/version behaviors through the runner/config layer.
- Environment-variable overrides (SUBFINDER_CONFIG, SUBFINDER_PROVIDER_CONFIG) affect config resolution at startup.
11. Configuration boundary
- Runtime behavior is driven by config.yaml and provider-config.yaml style inputs, with defaults under the user config directory.
- This cleanly separates the binary’s orchestration from provider credentials and operational tuning.
12. Testing and delivery
- Tests are concentrated around source adapters, rate limiting, and enumeration behavior.
- CI/release tooling in .github/workflows, plus goreleaser and Docker support, indicate this is shipped as a standalone binary and container image, not as a long-running service.
Main entry points: runner.go Runner · sources.go Providers · extractor.go Subscraping · config.go Config
Components 19
CLI entrypoints
Binary and library entry surfaces
- subfinder · Executable bootstrap for flags, config, and runner startup
cmd/subfinder/main.go - Example use · Library-style embedding of the runner
examples/main.go
Orchestration
Runner, config, and pipeline control
- Runner · Central orchestration of enumeration, validation, stats, and output
pkg/runner/runner.go - Options · CLI and runtime option parsing and normalization
pkg/runner/options.go - Config · Loads user and provider configuration with environment overrides
pkg/runner/config.go - Init · Prepares sources, resolvers, and execution state
pkg/runner/initialize.go - Enumerate · Drives domain input through passive collection and aggregation
pkg/runner/enumerate.go - Output · Formats results to stdout, files, JSONL, or list-mode outputs
pkg/runner/outputter.go - Stats · Tracks enumeration progress and execution summary
pkg/runner/stats.go
Passive collection
Source scraping and aggregation
- Passive · Top-level passive enumeration package
pkg/passive/passive.go - Subscraping · Common extraction framework, HTTP agent, and parsing utilities
pkg/subscraping/extractor.go - Providers · Registry and grouping of external source adapters
pkg/passive/sources.go - GitHub · Representative authenticated source adapter with token management
pkg/subscraping/sources/github/github.go - crt.sh · Representative passive certificate transparency source
pkg/subscraping/sources/crtsh/crtsh.go - Shodan · Representative API-backed source adapter
pkg/subscraping/sources/shodan/shodan.go
DNS validation
Active resolution and wildcard filtering
Delivery
Build, release, and packaging
- Release · Build and packaging automation for binaries and containers
.goreleaser.yml
Connections 23
- subfinder → Runner: starts · CLI hands control to the orchestration layer
- Example use → Runner: embeds · Example shows library-style reuse
- Runner → Options: reads · Runner interprets flags and execution mode
- Runner → Config: loads · Runtime configuration is resolved at startup
- Runner → Init: initializes · Prepares execution state and dependencies
- Runner → Enumerate: drives · Main enumeration pipeline
- Runner → Validate: optionally verifies · Active DNS validation is gated by options
- Runner → Output: formats · Results are emitted through the output layer
- Runner → Stats: tracks · Execution progress and summary are recorded
- Enumerate → Passive: collects via · Enumeration delegates to passive collection
- Passive → Subscraping: uses · Collector relies on shared scraping and extraction utilities
- Passive → Providers: selects · Source registry controls enabled providers
- Providers → GitHub: includes · Representative provider adapter
- Providers → crt.sh: includes · Representative provider adapter
- Providers → Shodan: includes · Representative provider adapter
- Subscraping → GitHub: adapts · Source adapters implement provider-specific requests and parsing
- Subscraping → crt.sh: adapts · Source adapters implement provider-specific requests and parsing
- Subscraping → Shodan: adapts · Source adapters implement provider-specific requests and parsing
- Validate → Resolve: uses · DNS validation resolves discovered subdomains
- Resolve → Resolvers: configures · Resolver client provides lookup behavior
- Config → Resolve: supplies · Resolver settings and custom lists flow from config
- Config → Providers: supplies · Provider credentials and limits are injected from provider config
- Release → subfinder: packages · Binary release and container build artifacts