# projectdiscovery/uncover architecture

> Architecture diagram of the GitHub repository projectdiscovery/uncover, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/projectdiscovery/uncover
- Repository: https://github.com/projectdiscovery/uncover

## Overview

1. **What this repo is**
   - `uncover` is a Go CLI and library for querying third-party internet-search APIs and normalizing the results into host/port/hostnames.
   - It is not a crawler or scanner itself; the core job is orchestration across external providers.

2. **Entry points**
   - The primary binary lives in `cmd/uncover/uncover.go`.
   - The package root `uncover.go` exposes the library-facing API used by `examples/main.go`.
   - Runtime behavior is driven by CLI flags, stdin, config files, and environment variables.

3. **Top-level orchestration layer**
   - The `runner/` package is the main application layer.
   - It parses options, prints banners/warnings, drives query execution, and sends results to the output writer.
   - This is the boundary between user input and provider-specific network calls.

4. **Provider abstraction**
   - `sources/provider.go`, `sources/session.go`, `sources/result.go`, `sources/keys.go`, and `sources/util.go` define the generic provider model.
   - Providers are selected by engine name, and each engine is implemented as an adapter with its own request/response mapping.
   - A provider session likely encapsulates authentication, retries, timeout, proxy support, and request execution.

5. **Engine adapters**
   - Each search engine has its own package under `sources/agent/<engine>/`.
   - Examples include Shodan, Censys, FOFA, Quake, Hunter, ZoomEye, Netlas, CriminalIP, PublicWWW, HunterHow, Google, Onyphe, Driftnet, DayDayMap, NerdyData, and Shodan InternetDB.
   - These packages isolate API-specific request formats, auth schemes, pagination, and result parsing.

6. **Input routing and engine selection**
   - The tool accepts either search dorks or IP/CIDR input.
   - Text queries default to a search provider such as Shodan; IP/CIDR input switches to InternetDB-style lookup by default.
   - Multiple engines can be targeted in one run, and per-engine queries can be specified independently.

7. **Credential and provider configuration**
   - Provider credentials are loaded from `$CONFIG/uncover/provider-config.yaml` or environment variables.
   - The repo supports multiple keys per provider and randomizes which one is used, which is important for rate distribution and failover.
   - This credential layer is a hard dependency for most engines; Shodan InternetDB is the notable exception.

8. **Execution concerns**
   - The main cross-cutting controls are timeout, retry, rate limiting, and proxy settings.
   - These controls apply at the session/network layer rather than inside each provider adapter, which keeps the engine packages focused on translation only.

9. **Result normalization and output**
   - Results are normalized into a shared shape and then rendered by `runner/output_writer.go`.
   - Output modes include plain text, JSONL, raw API output, and custom field formatting such as `ip`, `port`, or `host`.
   - This output layer is designed for piping into other tools, which is a key part of the architecture.

10. **Automation and pipeline integration**
    - The repo is built for composition with other ProjectDiscovery tools and shell pipelines.
    - Stdin/stdout support is first-class, so `uncover` acts as a query-and-normalize stage before tools like `httpx`, `naabu`, or `nuclei`.

11. **Testing boundaries**
    - `integration-tests/` and `testutils/` indicate real-provider integration coverage in addition to package tests like `sources/agent/shodanidb/shodan_test.go` and `sources/session_test.go`.
    - This matters architecturally because correctness depends on provider contracts, not just local logic.

12. **Build and release infrastructure**
    - The repo includes `Dockerfile`, `Makefile`, and `.goreleaser.yml`, plus GitHub Actions for build, lint, release, code scanning, Docker publishing, and provider integration tests.
    - These are delivery concerns rather than runtime components, but they reinforce that the project is shipped as a CLI binary and container image.

Main entry points: [`sources/provider.go`](https://github.com/projectdiscovery/uncover/blob/main/sources/provider.go) (Provider model), [`runner/runner.go`](https://github.com/projectdiscovery/uncover/blob/main/runner/runner.go) (Runner), [`sources/agent`](https://github.com/projectdiscovery/uncover/tree/main/sources/agent) (Search engines), [`integration-tests/integration-test.go`](https://github.com/projectdiscovery/uncover/blob/main/integration-tests/integration-test.go) (Integration tests)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entrypoints["Entry points"]
  node_cmd_uncover["uncover CLI<br/>binary entrypoint<br/>[uncover.go]"]
  node_lib_root["Library API<br/>package API<br/>[uncover.go]"]
  node_example_main["Example app<br/>example consumer<br/>[main.go]"]
end

subgraph group_runtime["Runtime"]
  node_runner["Runner<br/>orchestrator<br/>[runner.go]"]
  node_options["Options<br/>cli config<br/>[options.go]"]
  node_banners["Banners<br/>ui output<br/>[banners.go]"]
  node_writer["Output writer<br/>rendering<br/>[output_writer.go]"]
end

subgraph group_providers["Provider core"]
  node_provider_api["Provider model<br/>provider abstraction<br/>[provider.go]"]
  node_session["Session<br/>[session.go]"]
  node_result["Result model<br/>normalized schema<br/>[result.go]"]
  node_keys["Key config<br/>credential loading<br/>[keys.go]"]
  node_util["Utilities<br/>provider helpers<br/>[util.go]"]
end

subgraph group_adapters["Engine adapters"]
  node_search_adapters["Search engines<br/>engine adapters"]
  node_shodanidb["InternetDB<br/>engine adapter<br/>[shodan.go]"]
  node_shodan["Shodan<br/>engine adapter<br/>[shodan.go]"]
  node_censys["Censys<br/>engine adapter<br/>[censys.go]"]
  node_quake["Quake<br/>engine adapter<br/>[quake.go]"]
end

subgraph group_testing["Testing"]
  node_integration_tests["Integration tests<br/>provider contract tests"]
  node_testutils["Test utils<br/>test support<br/>[integration.go]"]
end

node_cmd_uncover -->|"starts"| node_runner
node_lib_root -->|"uses"| node_runner
node_example_main -->|"consumes"| node_lib_root
node_runner -->|"reads"| node_options
node_runner -->|"prints"| node_banners
node_runner -->|"dispatches"| node_provider_api
node_runner -->|"renders"| node_writer
node_provider_api -->|"uses"| node_session
node_provider_api -->|"normalizes"| node_result
node_provider_api -->|"loads"| node_keys
node_session -->|"relies on"| node_util
node_provider_api -->|"adapts"| node_search_adapters
node_provider_api -->|"routes IP lookups"| node_shodanidb
node_search_adapters -.->|"includes"| node_shodan
node_search_adapters -.->|"includes"| node_censys
node_search_adapters -.->|"includes"| node_quake
node_integration_tests -->|"covers"| node_session
node_integration_tests -->|"covers"| node_search_adapters
node_integration_tests -->|"uses"| node_testutils
node_writer -->|"formats"| node_result

click node_cmd_uncover "https://github.com/projectdiscovery/uncover/blob/main/cmd/uncover/uncover.go"
click node_lib_root "https://github.com/projectdiscovery/uncover/blob/main/uncover.go"
click node_example_main "https://github.com/projectdiscovery/uncover/blob/main/examples/main.go"
click node_runner "https://github.com/projectdiscovery/uncover/blob/main/runner/runner.go"
click node_options "https://github.com/projectdiscovery/uncover/blob/main/runner/options.go"
click node_banners "https://github.com/projectdiscovery/uncover/blob/main/runner/banners.go"
click node_writer "https://github.com/projectdiscovery/uncover/blob/main/runner/output_writer.go"
click node_provider_api "https://github.com/projectdiscovery/uncover/blob/main/sources/provider.go"
click node_session "https://github.com/projectdiscovery/uncover/blob/main/sources/session.go"
click node_result "https://github.com/projectdiscovery/uncover/blob/main/sources/result.go"
click node_keys "https://github.com/projectdiscovery/uncover/blob/main/sources/keys.go"
click node_util "https://github.com/projectdiscovery/uncover/blob/main/sources/util.go"
click node_search_adapters "https://github.com/projectdiscovery/uncover/tree/main/sources/agent"
click node_shodanidb "https://github.com/projectdiscovery/uncover/blob/main/sources/agent/shodanidb/shodan.go"
click node_shodan "https://github.com/projectdiscovery/uncover/blob/main/sources/agent/shodan/shodan.go"
click node_censys "https://github.com/projectdiscovery/uncover/blob/main/sources/agent/censys/censys.go"
click node_quake "https://github.com/projectdiscovery/uncover/blob/main/sources/agent/quake/quake.go"
click node_integration_tests "https://github.com/projectdiscovery/uncover/blob/main/integration-tests/integration-test.go"
click node_testutils "https://github.com/projectdiscovery/uncover/blob/main/testutils/integration.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cmd_uncover,node_lib_root,node_example_main toneBlue
class node_runner,node_options,node_banners,node_writer toneAmber
class node_provider_api,node_session,node_result,node_keys,node_util toneMint
class node_search_adapters,node_shodanidb,node_shodan,node_censys,node_quake toneRose
class node_integration_tests,node_testutils toneIndigo
```

## Components

### Entry points

CLI and library surfaces

- [uncover CLI](https://github.com/projectdiscovery/uncover/blob/main/cmd/uncover/uncover.go): binary entrypoint · Main command-line entrypoint · `cmd/uncover/uncover.go`
- [Library API](https://github.com/projectdiscovery/uncover/blob/main/uncover.go): package API · Public package used by examples and external callers · `uncover.go`
- [Example app](https://github.com/projectdiscovery/uncover/blob/main/examples/main.go): example consumer · Demonstrates library usage · `examples/main.go`

### Runtime

Orchestration, selection, and rendering

- [Runner](https://github.com/projectdiscovery/uncover/blob/main/runner/runner.go): orchestrator · Parses options, selects engines, drives query execution · `runner/runner.go`
- [Options](https://github.com/projectdiscovery/uncover/blob/main/runner/options.go): cli config · Flags, input routing, and runtime settings · `runner/options.go`
- [Banners](https://github.com/projectdiscovery/uncover/blob/main/runner/banners.go): ui output · Startup messages and warnings · `runner/banners.go`
- [Output writer](https://github.com/projectdiscovery/uncover/blob/main/runner/output_writer.go): rendering · Formats normalized results for stdout and piping · `runner/output_writer.go`

### Provider core

Shared provider/session/result abstractions

- [Provider model](https://github.com/projectdiscovery/uncover/blob/main/sources/provider.go): provider abstraction · Generic engine interface and selection model · `sources/provider.go`
- [Session](https://github.com/projectdiscovery/uncover/blob/main/sources/session.go): network session · Shared execution controls such as auth, retry, timeout, and proxy · `sources/session.go`
- [Result model](https://github.com/projectdiscovery/uncover/blob/main/sources/result.go): normalized schema · Shared output shape for hosts, ports, and hostnames · `sources/result.go`
- [Key config](https://github.com/projectdiscovery/uncover/blob/main/sources/keys.go): credential loading · Loads provider credentials and chooses among multiple keys · `sources/keys.go`
- [Utilities](https://github.com/projectdiscovery/uncover/blob/main/sources/util.go): provider helpers · Common provider-side helper logic · `sources/util.go`

### Engine adapters

Search engine integrations and request/response mapping

- [Search engines](https://github.com/projectdiscovery/uncover/tree/main/sources/agent): engine adapters · Adapters for API-backed search providers such as Shodan, Censys, FOFA, Quake, Hunter, ZoomEye, Netlas, CriminalIP, PublicWWW, Google, Onyphe, Driftnet, DayDayMap, NerdyData, BinaryEdge, Greynoise, and Odin · `sources/agent`
- [InternetDB](https://github.com/projectdiscovery/uncover/blob/main/sources/agent/shodanidb/shodan.go): engine adapter · IP/CIDR lookup path for Shodan InternetDB-style results · `sources/agent/shodanidb/shodan.go`
- [Shodan](https://github.com/projectdiscovery/uncover/blob/main/sources/agent/shodan/shodan.go): engine adapter · Search engine adapter with provider-specific mapping · `sources/agent/shodan/shodan.go`
- [Censys](https://github.com/projectdiscovery/uncover/blob/main/sources/agent/censys/censys.go): engine adapter · Search engine adapter · `sources/agent/censys/censys.go`
- [Quake](https://github.com/projectdiscovery/uncover/blob/main/sources/agent/quake/quake.go): engine adapter · Search engine adapter · `sources/agent/quake/quake.go`

### Testing

Integration and provider contract coverage

- [Integration tests](https://github.com/projectdiscovery/uncover/blob/main/integration-tests/integration-test.go): provider contract tests · Real-provider and session coverage · `integration-tests/integration-test.go`
- [Test utils](https://github.com/projectdiscovery/uncover/blob/main/testutils/integration.go): test support · Shared helpers for integration coverage · `testutils/integration.go`

## Connections

- uncover CLI → Runner: starts · CLI delegates to the runtime orchestrator
- Library API → Runner: uses · Library API routes work into the runner
- Example app → Library API: consumes · Example shows library-facing usage
- Runner → Options: reads · Runner interprets flags, input mode, and engine selection
- Runner → Banners: prints · Emits warnings and startup banners
- Runner → Provider model: dispatches · Coordinates provider selection and execution
- Runner → Output writer: renders · Sends normalized results to output formatting
- Provider model → Session: uses · Provider executions are mediated by shared sessions
- Provider model → Result model: normalizes · Providers map API responses into the shared result shape
- Provider model → Key config: loads · Credentials are resolved through provider key config
- Session → Utilities: relies on · Session execution uses shared helper logic
- Provider model → Search engines: adapts · Engine adapters implement provider-specific request and parsing logic
- Provider model → InternetDB: routes IP lookups · IP/CIDR input can bypass search engines for InternetDB-style lookup
- Search engines → Shodan: includes · Representative search provider adapter
- Search engines → Censys: includes · Representative search provider adapter
- Search engines → Quake: includes · Representative search provider adapter
- Integration tests → Session: covers · Integration tests validate session behavior against real providers
- Integration tests → Search engines: covers · Provider contract coverage exercises adapter implementations
- Integration tests → Test utils: uses · Shared test helpers support integration execution
- Output writer → Result model: formats · Output writer renders normalized records