projectdiscovery/uncover

1. What this repo is

- uncover is a Go CLI and library for querying third-party internet-search APIs and normalizing the results into host/port/hostnames.

- It is not a crawler or scanner itself; the core job is orchestration across external providers.

2. Entry points

- The primary binary lives in cmd/uncover/uncover.go.

- The package root uncover.go exposes the library-facing API used by examples/main.go.

- Runtime behavior is driven by CLI flags, stdin, config files, and environment variables.

3. Top-level orchestration layer

- The runner/ package is the main application layer.

- It parses options, prints banners/warnings, drives query execution, and sends results to the output writer.

- This is the boundary between user input and provider-specific network calls.

4. Provider abstraction

- sources/provider.go, sources/session.go, sources/result.go, sources/keys.go, and sources/util.go define the generic provider model.

- Providers are selected by engine name, and each engine is implemented as an adapter with its own request/response mapping.

- A provider session likely encapsulates authentication, retries, timeout, proxy support, and request execution.

5. Engine adapters

- Each search engine has its own package under sources/agent/<engine>/.

- Examples include Shodan, Censys, FOFA, Quake, Hunter, ZoomEye, Netlas, CriminalIP, PublicWWW, HunterHow, Google, Onyphe, Driftnet, DayDayMap, NerdyData, and Shodan InternetDB.

- These packages isolate API-specific request formats, auth schemes, pagination, and result parsing.

6. Input routing and engine selection

- The tool accepts either search dorks or IP/CIDR input.

- Text queries default to a search provider such as Shodan; IP/CIDR input switches to InternetDB-style lookup by default.

- Multiple engines can be targeted in one run, and per-engine queries can be specified independently.

7. Credential and provider configuration

- Provider credentials are loaded from $CONFIG/uncover/provider-config.yaml or environment variables.

- The repo supports multiple keys per provider and randomizes which one is used, which is important for rate distribution and failover.

- This credential layer is a hard dependency for most engines; Shodan InternetDB is the notable exception.

8. Execution concerns

- The main cross-cutting controls are timeout, retry, rate limiting, and proxy settings.

- These controls apply at the session/network layer rather than inside each provider adapter, which keeps the engine packages focused on translation only.

9. Result normalization and output

- Results are normalized into a shared shape and then rendered by runner/output_writer.go.

- Output modes include plain text, JSONL, raw API output, and custom field formatting such as ip, port, or host.

- This output layer is designed for piping into other tools, which is a key part of the architecture.

10. Automation and pipeline integration

- The repo is built for composition with other ProjectDiscovery tools and shell pipelines.

- Stdin/stdout support is first-class, so uncover acts as a query-and-normalize stage before tools like httpx, naabu, or nuclei.

11. Testing boundaries

- integration-tests/ and testutils/ indicate real-provider integration coverage in addition to package tests like sources/agent/shodanidb/shodan_test.go and sources/session_test.go.

- This matters architecturally because correctness depends on provider contracts, not just local logic.

12. Build and release infrastructure

- The repo includes Dockerfile, Makefile, and .goreleaser.yml, plus GitHub Actions for build, lint, release, code scanning, Docker publishing, and provider integration tests.

- These are delivery concerns rather than runtime components, but they reinforce that the project is shipped as a CLI binary and container image.