# projectdiscovery/urlfinder architecture

> Architecture diagram of the GitHub repository projectdiscovery/urlfinder, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/projectdiscovery/urlfinder
- Repository: https://github.com/projectdiscovery/urlfinder

## Overview

URLFinder is a passive URL-discovery CLI. The main workflow accepts domains from flags or stdin, selects passive providers, runs their searches concurrently through a shared HTTP session, extracts URLs, applies scope and output handling, and emits results. The graph also preserves documented filtering, JSONL/file output, source selection, and statistics capabilities. Some runner and provider wiring is unsampled; edges are limited to relationships visible in excerpts, the source index, or README documentation.

Main entry points: [`pkg/agent/agent.go`](https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/agent/agent.go) (Source coordinator), [`pkg/session/session.go`](https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/session/session.go) (HTTP session), [`internal/runner/enumerate.go`](https://github.com/projectdiscovery/urlfinder/blob/dev/internal/runner/enumerate.go) (Query enumeration), [`pkg/agent/registry.go`](https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/agent/registry.go) (Source registry)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["CLI and orchestration"]
  node_cli["CLI entry<br/>[main.go]"]
  node_options["Options and config<br/>[options.go]"]
  node_runner["Enumeration runner<br/>[runner.go]"]
  node_enumerator["Query enumeration<br/>[enumerate.go]"]
end

subgraph group_sources["Passive providers"]
  node_registry["Source registry<br/>[registry.go]"]
  node_agent["Source coordinator<br/>[agent.go]"]
  node_alienvault["AlienVault<br/>[alienvault.go]"]
  node_commoncrawl["Common Crawl<br/>[commoncrawl.go]"]
  node_urlscan["urlscan<br/>[urlscan.go]"]
  node_virustotal["VirusTotal<br/>[virustotal.go]"]
  node_wayback["Wayback Archive<br/>[waybackarchive.go]"]
  node_passive_apis(("Passive data services"))
end

subgraph group_runtime["Discovery runtime"]
  node_session["HTTP session<br/>[session.go]"]
  node_extractor["URL extractor<br/>[regex_extractor.go]"]
end

subgraph group_results["Result processing"]
  node_scope["Scope validation<br/>[scope.go]"]
  node_extensions["Extension filters<br/>[extensions.go]"]
  node_resolver["URL resolution<br/>[resolve.go]"]
  node_output["Output formatting<br/>[outputter.go]"]
  node_statistics["Source statistics<br/>[stats.go]"]
end

node_user(("Operator"))
node_destination(("Stdout or files"))

node_user -->|"provides inputs"| node_cli
node_cli -->|"parses flags"| node_options
node_cli -->|"starts enumeration"| node_runner
node_options -->|"configures"| node_runner
node_runner -->|"enumerates targets"| node_enumerator
node_runner -->|"validates scope"| node_scope
node_enumerator -->|"dispatches queries"| node_agent
node_enumerator -->|"uses resolution"| node_resolver
node_options -->|"selects sources"| node_agent
node_options -->|"configures filters"| node_extensions
node_agent -->|"selects providers"| node_registry
node_registry -->|"registers"| node_alienvault
node_registry -->|"registers"| node_commoncrawl
node_registry -->|"registers"| node_urlscan
node_registry -->|"registers"| node_virustotal
node_registry -->|"registers"| node_wayback
node_agent -->|"creates session"| node_session
node_agent -->|"runs provider"| node_alienvault
node_agent -->|"runs provider"| node_commoncrawl
node_agent -->|"runs provider"| node_urlscan
node_agent -->|"runs provider"| node_virustotal
node_agent -->|"runs provider"| node_wayback
node_alienvault -->|"requests data"| node_session
node_commoncrawl -->|"requests data"| node_session
node_urlscan -->|"requests data"| node_session
node_virustotal -->|"requests data"| node_session
node_wayback -->|"requests data"| node_session
node_session -->|"creates extractor"| node_extractor
node_session -.->|"sends HTTP requests"| node_passive_apis
node_enumerator -->|"formats results"| node_output
node_enumerator -->|"reports statistics"| node_statistics
node_output -->|"writes output"| node_destination
node_user -->|"sets filters"| node_options
node_enumerator -.->|"filters extensions"| node_extensions

click node_cli "https://github.com/projectdiscovery/urlfinder/blob/dev/cmd/urlfinder/main.go"
click node_options "https://github.com/projectdiscovery/urlfinder/blob/dev/internal/runner/options.go"
click node_runner "https://github.com/projectdiscovery/urlfinder/blob/dev/internal/runner/runner.go"
click node_enumerator "https://github.com/projectdiscovery/urlfinder/blob/dev/internal/runner/enumerate.go"
click node_registry "https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/agent/registry.go"
click node_agent "https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/agent/agent.go"
click node_alienvault "https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/source/alienvault/alienvault.go"
click node_commoncrawl "https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/source/commoncrawl/commoncrawl.go"
click node_urlscan "https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/source/urlscan/urlscan.go"
click node_virustotal "https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/source/virustotal/virustotal.go"
click node_wayback "https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/source/waybackarchive/waybackarchive.go"
click node_session "https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/session/session.go"
click node_extractor "https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/extractor/regex_extractor.go"
click node_scope "https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/utils/scope/scope.go"
click node_extensions "https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/utils/extensions/extensions.go"
click node_resolver "https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/resolve/resolve.go"
click node_output "https://github.com/projectdiscovery/urlfinder/blob/dev/internal/runner/outputter.go"
click node_statistics "https://github.com/projectdiscovery/urlfinder/blob/dev/internal/runner/stats.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_options,node_runner,node_enumerator toneBlue
class node_registry,node_agent,node_alienvault,node_commoncrawl,node_urlscan,node_virustotal,node_wayback,node_passive_apis toneAmber
class node_session,node_extractor toneMint
class node_scope,node_extensions,node_resolver,node_output,node_statistics toneRose
class node_user,node_destination toneIndigo
```

## Components

### CLI and orchestration

- [CLI entry](https://github.com/projectdiscovery/urlfinder/blob/dev/cmd/urlfinder/main.go): `cmd/urlfinder/main.go`
- [Options and config](https://github.com/projectdiscovery/urlfinder/blob/dev/internal/runner/options.go): `internal/runner/options.go`
- [Enumeration runner](https://github.com/projectdiscovery/urlfinder/blob/dev/internal/runner/runner.go): `internal/runner/runner.go`
- [Query enumeration](https://github.com/projectdiscovery/urlfinder/blob/dev/internal/runner/enumerate.go): `internal/runner/enumerate.go`

### Passive providers

- [Source registry](https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/agent/registry.go): `pkg/agent/registry.go`
- [Source coordinator](https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/agent/agent.go): `pkg/agent/agent.go`
- [AlienVault](https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/source/alienvault/alienvault.go): `pkg/source/alienvault/alienvault.go`
- [Common Crawl](https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/source/commoncrawl/commoncrawl.go): `pkg/source/commoncrawl/commoncrawl.go`
- [urlscan](https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/source/urlscan/urlscan.go): `pkg/source/urlscan/urlscan.go`
- [VirusTotal](https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/source/virustotal/virustotal.go): `pkg/source/virustotal/virustotal.go`
- [Wayback Archive](https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/source/waybackarchive/waybackarchive.go): `pkg/source/waybackarchive/waybackarchive.go`
- **Passive data services**

### Discovery runtime

- [HTTP session](https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/session/session.go): `pkg/session/session.go`
- [URL extractor](https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/extractor/regex_extractor.go): `pkg/extractor/regex_extractor.go`

### Result processing

- [Scope validation](https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/utils/scope/scope.go): `pkg/utils/scope/scope.go`
- [Extension filters](https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/utils/extensions/extensions.go): `pkg/utils/extensions/extensions.go`
- [URL resolution](https://github.com/projectdiscovery/urlfinder/blob/dev/pkg/resolve/resolve.go): `pkg/resolve/resolve.go`
- [Output formatting](https://github.com/projectdiscovery/urlfinder/blob/dev/internal/runner/outputter.go): `internal/runner/outputter.go`
- [Source statistics](https://github.com/projectdiscovery/urlfinder/blob/dev/internal/runner/stats.go): `internal/runner/stats.go`

### Other components

- **Operator**
- **Stdout or files**

## Connections

- Operator → CLI entry: provides inputs (evidence: `README.md`)
- CLI entry → Options and config: parses flags (evidence: `cmd/urlfinder/main.go`)
- CLI entry → Enumeration runner: starts enumeration (evidence: `cmd/urlfinder/main.go`)
- Options and config → Enumeration runner: configures (evidence: `internal/runner/runner.go`)
- Enumeration runner → Query enumeration: enumerates targets (evidence: `internal/runner/runner.go`)
- Enumeration runner → Scope validation: validates scope (evidence: `internal/runner/runner.go`)
- Query enumeration → Source coordinator: dispatches queries (evidence: `internal/runner/enumerate.go`)
- Query enumeration → URL resolution: uses resolution (evidence: `internal/runner/enumerate.go`)
- Options and config → Source coordinator: selects sources (evidence: `internal/runner/options.go`)
- Options and config → Extension filters: configures filters (evidence: `internal/runner/options.go`)
- Source coordinator → Source registry: selects providers (evidence: `pkg/agent/agent.go`)
- Source registry → AlienVault: registers (evidence: `pkg/agent/registry.go`)
- Source registry → Common Crawl: registers (evidence: `pkg/agent/registry.go`)
- Source registry → urlscan: registers (evidence: `pkg/agent/registry.go`)
- Source registry → VirusTotal: registers (evidence: `pkg/agent/registry.go`)
- Source registry → Wayback Archive: registers (evidence: `pkg/agent/registry.go`)
- Source coordinator → HTTP session: creates session (evidence: `pkg/agent/agent.go`)
- Source coordinator → AlienVault: runs provider (evidence: `pkg/agent/agent.go`)
- Source coordinator → Common Crawl: runs provider (evidence: `pkg/agent/agent.go`)
- Source coordinator → urlscan: runs provider (evidence: `pkg/agent/agent.go`)
- Source coordinator → VirusTotal: runs provider (evidence: `pkg/agent/agent.go`)
- Source coordinator → Wayback Archive: runs provider (evidence: `pkg/agent/agent.go`)
- AlienVault → HTTP session: requests data (evidence: `pkg/source/alienvault/alienvault.go`)
- Common Crawl → HTTP session: requests data (evidence: `pkg/source/commoncrawl/commoncrawl.go`)
- urlscan → HTTP session: requests data (evidence: `pkg/source/urlscan/urlscan.go`)
- VirusTotal → HTTP session: requests data (evidence: `pkg/source/virustotal/virustotal.go`)
- Wayback Archive → HTTP session: requests data (evidence: `pkg/source/waybackarchive/waybackarchive.go`)
- HTTP session → URL extractor: creates extractor (evidence: `pkg/session/session.go`)
- HTTP session → Passive data services: sends HTTP requests
- Query enumeration → Output formatting: formats results (evidence: `internal/runner/enumerate.go`)
- Query enumeration → Source statistics: reports statistics (evidence: `internal/runner/enumerate.go`)
- Output formatting → Stdout or files: writes output (evidence: `README.md`)
- Operator → Options and config: sets filters (evidence: `README.md`)
- Query enumeration → Extension filters: filters extensions