# josh0xa/darkdump architecture

> Architecture diagram of the GitHub repository josh0xa/darkdump, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/josh0xa/darkdump
- Repository: https://github.com/josh0xa/darkdump

## Overview

Darkdump is an OSINT search and investigation tool with CLI and local-web entry points. The main workflow submits a query to one of six search engines, checks results against Ahmia’s abuse blacklist, optionally scrapes result sites, and returns or exports findings. A separate breach-intelligence mode generates related searches and classifies/extracts artefacts from results. Tor proxying, deduplication, image collection, and live web streaming are supporting capabilities. The map combines README-documented features with sampled code; several internal relationships and the CLI’s detailed control flow are not shown in the excerpts.

Main entry points: [`darkdump-web/app.py`](https://github.com/josh0xa/darkdump/blob/main/darkdump-web/app.py) (Web interface), [`darkdump.py`](https://github.com/josh0xa/darkdump/blob/main/darkdump.py) (CLI), [`darkdump-web/templates/index.html`](https://github.com/josh0xa/darkdump/blob/main/darkdump-web/templates/index.html) (Browser UI), [`headers/agents.py`](https://github.com/josh0xa/darkdump/blob/main/headers/agents.py) (User-agent pool)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["User interfaces"]
  node_cli["CLI<br/>[darkdump.py]"]
  node_web["Web interface<br/>[app.py]"]
  node_webui["Browser UI<br/>[index.html]"]
end

subgraph group_search["Search and safeguards"]
  node_engine_adapter["Search engines<br/>[darkdump.py]"]
  node_blacklist["Abuse blacklist<br/>[darkdump.py]"]
end

subgraph group_analysis["Investigation"]
  node_scraper["Site scraping<br/>[darkdump.py]"]
  node_breach["Breach intelligence<br/>[darkdump.py]"]
  node_dedupe["Result deduplication<br/>[darkdump.py]"]
end

subgraph group_support["Transport and output"]
  node_export["Result export<br/>[darkdump.py]"]
  node_stream["Live event stream<br/>[app.py]"]
  node_agents["User-agent pool<br/>[agents.py]"]
end

node_researcher(("Researcher"))
node_ahmia["Ahmia blacklist"]
node_tor["Tor service"]

node_researcher -->|"runs search"| node_cli
node_researcher -->|"enters query"| node_webui
node_webui -->|"submits options"| node_web
node_cli -->|"searches engines"| node_engine_adapter
node_web -->|"calls fetchers"| node_engine_adapter
node_engine_adapter -->|"filters results"| node_blacklist
node_blacklist -->|"checks blacklist"| node_ahmia
node_cli -->|"optionally scrapes"| node_scraper
node_web -->|"optionally scrapes"| node_scraper
node_cli -->|"runs scan"| node_breach
node_web -->|"runs scan"| node_breach
node_web -->|"deduplicates results"| node_dedupe
node_cli -->|"deduplicates results"| node_dedupe
node_cli -->|"saves results"| node_export
node_web -->|"exports results"| node_export
node_web -->|"streams events"| node_stream
node_stream -->|"updates display"| node_webui
node_cli -->|"proxies requests"| node_tor
node_web -->|"checks and proxies"| node_tor
node_web -->|"selects user agent"| node_agents
node_cli -.->|"uses user agents"| node_agents

click node_cli "https://github.com/josh0xa/darkdump/blob/main/darkdump.py"
click node_web "https://github.com/josh0xa/darkdump/blob/main/darkdump-web/app.py"
click node_webui "https://github.com/josh0xa/darkdump/blob/main/darkdump-web/templates/index.html"
click node_engine_adapter "https://github.com/josh0xa/darkdump/blob/main/darkdump.py"
click node_blacklist "https://github.com/josh0xa/darkdump/blob/main/darkdump.py"
click node_scraper "https://github.com/josh0xa/darkdump/blob/main/darkdump.py"
click node_breach "https://github.com/josh0xa/darkdump/blob/main/darkdump.py"
click node_dedupe "https://github.com/josh0xa/darkdump/blob/main/darkdump.py"
click node_export "https://github.com/josh0xa/darkdump/blob/main/darkdump.py"
click node_stream "https://github.com/josh0xa/darkdump/blob/main/darkdump-web/app.py"
click node_agents "https://github.com/josh0xa/darkdump/blob/main/headers/agents.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_web,node_webui toneBlue
class node_engine_adapter,node_blacklist toneAmber
class node_scraper,node_breach,node_dedupe toneMint
class node_export,node_stream,node_agents toneRose
class node_researcher,node_ahmia,node_tor toneIndigo
```

## Components

### User interfaces

- [CLI](https://github.com/josh0xa/darkdump/blob/main/darkdump.py): `darkdump.py`
- [Web interface](https://github.com/josh0xa/darkdump/blob/main/darkdump-web/app.py): `darkdump-web/app.py`
- [Browser UI](https://github.com/josh0xa/darkdump/blob/main/darkdump-web/templates/index.html): `darkdump-web/templates/index.html`

### Search and safeguards

- [Search engines](https://github.com/josh0xa/darkdump/blob/main/darkdump.py): `darkdump.py`
- [Abuse blacklist](https://github.com/josh0xa/darkdump/blob/main/darkdump.py): `darkdump.py`

### Investigation

- [Site scraping](https://github.com/josh0xa/darkdump/blob/main/darkdump.py): `darkdump.py`
- [Breach intelligence](https://github.com/josh0xa/darkdump/blob/main/darkdump.py): `darkdump.py`
- [Result deduplication](https://github.com/josh0xa/darkdump/blob/main/darkdump.py): `darkdump.py`

### Transport and output

- [Result export](https://github.com/josh0xa/darkdump/blob/main/darkdump.py): `darkdump.py`
- [Live event stream](https://github.com/josh0xa/darkdump/blob/main/darkdump-web/app.py): `darkdump-web/app.py`
- [User-agent pool](https://github.com/josh0xa/darkdump/blob/main/headers/agents.py): `headers/agents.py`

### Other components

- **Researcher**
- **Ahmia blacklist**
- **Tor service**

## Connections

- Researcher → CLI: runs search (evidence: `README.md`)
- Researcher → Browser UI: enters query (evidence: `README.md`)
- Browser UI → Web interface: submits options (evidence: `darkdump-web/app.py`)
- CLI → Search engines: searches engines (evidence: `README.md`)
- Web interface → Search engines: calls fetchers (evidence: `darkdump-web/app.py`)
- Search engines → Abuse blacklist: filters results (evidence: `README.md`)
- Abuse blacklist → Ahmia blacklist: checks blacklist (evidence: `README.md`)
- CLI → Site scraping: optionally scrapes (evidence: `README.md`)
- Web interface → Site scraping: optionally scrapes (evidence: `darkdump-web/app.py`)
- CLI → Breach intelligence: runs scan (evidence: `README.md`)
- Web interface → Breach intelligence: runs scan (evidence: `darkdump-web/app.py`)
- Web interface → Result deduplication: deduplicates results (evidence: `darkdump-web/app.py`)
- CLI → Result deduplication: deduplicates results (evidence: `README.md`)
- CLI → Result export: saves results (evidence: `README.md`)
- Web interface → Result export: exports results (evidence: `README.md`)
- Web interface → Live event stream: streams events (evidence: `darkdump-web/app.py`)
- Live event stream → Browser UI: updates display (evidence: `README.md`)
- CLI → Tor service: proxies requests (evidence: `README.md`)
- Web interface → Tor service: checks and proxies (evidence: `darkdump-web/app.py`)
- Web interface → User-agent pool: selects user agent (evidence: `darkdump-web/app.py`)
- CLI → User-agent pool: uses user agents (evidence: `darkdump.py`)
