How projectdiscovery/urlfinder works
URLFinder is a passive URL-discovery CLI. The main workflow accepts domains from flags or stdin, selects passive providers, runs their searches concurrently through a shared HTTP session, extracts URLs, applies scope and output handling, and emits results. The graph also preserves documented filtering, JSONL/file output, source selection, and statistics capabilities. Some runner and provider wiring is unsampled; edges are limited to relationships visible in excerpts, the source index, or README documentation.
Main entry points: agent.go Source coordinator · session.go HTTP session · enumerate.go Query enumeration · registry.go Source registry
Components 21
CLI and orchestration
- CLI entry
cmd/urlfinder/main.go - Options and config
internal/runner/options.go - Enumeration runner
internal/runner/runner.go - Query enumeration
internal/runner/enumerate.go
Passive providers
- Source registry
pkg/agent/registry.go - Source coordinator
pkg/agent/agent.go - AlienVault
pkg/source/alienvault/alienvault.go - Common Crawl
pkg/source/commoncrawl/commoncrawl.go - urlscan
pkg/source/urlscan/urlscan.go - VirusTotal
pkg/source/virustotal/virustotal.go - Wayback Archive
pkg/source/waybackarchive/waybackarchive.go - Passive data services
Discovery runtime
- HTTP session
pkg/session/session.go - URL extractor
pkg/extractor/regex_extractor.go
Result processing
- Scope validation
pkg/utils/scope/scope.go - Extension filters
pkg/utils/extensions/extensions.go - URL resolution
pkg/resolve/resolve.go - Output formatting
internal/runner/outputter.go - Source statistics
internal/runner/stats.go
Other components
- Operator
- Stdout or files
Connections 34
- Operator → CLI entry: provides inputs
README.md - CLI entry → Options and config: parses flags
cmd/urlfinder/main.go - CLI entry → Enumeration runner: starts enumeration
cmd/urlfinder/main.go - Options and config → Enumeration runner: configures
internal/runner/runner.go - Enumeration runner → Query enumeration: enumerates targets
internal/runner/runner.go - Enumeration runner → Scope validation: validates scope
internal/runner/runner.go - Query enumeration → Source coordinator: dispatches queries
internal/runner/enumerate.go - Query enumeration → URL resolution: uses resolution
internal/runner/enumerate.go - Options and config → Source coordinator: selects sources
internal/runner/options.go - Options and config → Extension filters: configures filters
internal/runner/options.go - Source coordinator → Source registry: selects providers
pkg/agent/agent.go - Source registry → AlienVault: registers
pkg/agent/registry.go - Source registry → Common Crawl: registers
pkg/agent/registry.go - Source registry → urlscan: registers
pkg/agent/registry.go - Source registry → VirusTotal: registers
pkg/agent/registry.go - Source registry → Wayback Archive: registers
pkg/agent/registry.go - Source coordinator → HTTP session: creates session
pkg/agent/agent.go - Source coordinator → AlienVault: runs provider
pkg/agent/agent.go - Source coordinator → Common Crawl: runs provider
pkg/agent/agent.go - Source coordinator → urlscan: runs provider
pkg/agent/agent.go - Source coordinator → VirusTotal: runs provider
pkg/agent/agent.go - Source coordinator → Wayback Archive: runs provider
pkg/agent/agent.go - AlienVault → HTTP session: requests data
pkg/source/alienvault/alienvault.go - Common Crawl → HTTP session: requests data
pkg/source/commoncrawl/commoncrawl.go - urlscan → HTTP session: requests data
pkg/source/urlscan/urlscan.go - VirusTotal → HTTP session: requests data
pkg/source/virustotal/virustotal.go - Wayback Archive → HTTP session: requests data
pkg/source/waybackarchive/waybackarchive.go - HTTP session → URL extractor: creates extractor
pkg/session/session.go - HTTP session → Passive data services: sends HTTP requests
- Query enumeration → Output formatting: formats results
internal/runner/enumerate.go - Query enumeration → Source statistics: reports statistics
internal/runner/enumerate.go - Output formatting → Stdout or files: writes output
README.md - Operator → Options and config: sets filters
README.md - Query enumeration → Extension filters: filters extensions