# smicallef/spiderfoot architecture

> Architecture diagram of the GitHub repository smicallef/spiderfoot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/smicallef/spiderfoot
- Repository: https://github.com/smicallef/spiderfoot

## Overview

SpiderFoot automates OSINT reconnaissance through either its web UI or CLI. A user defines a target and scan configuration; scan control initializes selected plugins, which gather and publish findings for other plugins to consume. Findings are persisted for review, correlation, visualization and export. The graph preserves documented capability branches such as threat intelligence, discovery, dark-web search and cloud-bucket checks. Some branches are represented by their module directory because their implementations were not sampled; those relationships are documented at the capability level rather than inferred from module names.

Main entry points: [`spiderfoot/plugin.py`](https://github.com/smicallef/spiderfoot/blob/master/spiderfoot/plugin.py) (Plugin runtime), [`sfwebui.py`](https://github.com/smicallef/spiderfoot/blob/master/sfwebui.py) (Web interface), [`sfscan.py`](https://github.com/smicallef/spiderfoot/blob/master/sfscan.py) (Scan control), [`spiderfoot/db.py`](https://github.com/smicallef/spiderfoot/blob/master/spiderfoot/db.py) (Scan database)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["User interfaces"]
  node_web["Web interface<br/>[sfwebui.py]"]
  node_cli["Command line<br/>[sfcli.py]"]
  node_entry["Application entry<br/>[sf.py]"]
end

subgraph group_scan["Scan execution"]
  node_scancontrol["Scan control<br/>[sfscan.py]"]
  node_common["SpiderFoot services<br/>[sflib.py]"]
  node_plugin["Plugin runtime<br/>[plugin.py]"]
  node_events["Event model<br/>[event.py]"]
  node_pool["Thread pool<br/>[threadpool.py]"]
end

subgraph group_modules["Discovery modules"]
  node_discovery["Discovery and enrichment"]
  node_threat["Threat intelligence"]
  node_webdata["Web and social discovery"]
  node_darkweb["Dark-web search"]
  node_cloud["Cloud bucket checks"]
end

subgraph group_analysis["Analysis and results"]
  node_storage[("Scan database<br/>[db.py]")]
  node_correlation["Correlation engine<br/>[correlation.py]"]
  node_results["Results and visualization<br/>[viz.js]"]
end

node_user(("Analyst"))

node_user -->|"configures scan"| node_web
node_user -->|"runs scans"| node_cli
node_web -->|"starts scan"| node_scancontrol
node_entry -->|"starts web UI"| node_web
node_entry -->|"starts scanning"| node_scancontrol
node_scancontrol -->|"uses services"| node_common
node_scancontrol -->|"initializes plugins"| node_plugin
node_plugin -->|"submits tasks"| node_pool
node_plugin -->|"publishes events"| node_events
node_plugin -.->|"dispatches events"| node_discovery
node_plugin -.->|"dispatches events"| node_threat
node_plugin -.->|"dispatches events"| node_webdata
node_plugin -.->|"dispatches events"| node_darkweb
node_plugin -.->|"dispatches events"| node_cloud
node_discovery -.->|"publishes findings"| node_plugin
node_threat -.->|"publishes findings"| node_plugin
node_webdata -.->|"publishes findings"| node_plugin
node_darkweb -->|"publishes findings"| node_plugin
node_cloud -.->|"publishes findings"| node_plugin
node_scancontrol -.->|"stores scan data"| node_storage
node_correlation -->|"queries events"| node_storage
node_web -.->|"reads scan data"| node_storage
node_web -->|"shows visualizations"| node_results

click node_web "https://github.com/smicallef/spiderfoot/blob/master/sfwebui.py"
click node_cli "https://github.com/smicallef/spiderfoot/blob/master/sfcli.py"
click node_entry "https://github.com/smicallef/spiderfoot/blob/master/sf.py"
click node_scancontrol "https://github.com/smicallef/spiderfoot/blob/master/sfscan.py"
click node_common "https://github.com/smicallef/spiderfoot/blob/master/sflib.py"
click node_plugin "https://github.com/smicallef/spiderfoot/blob/master/spiderfoot/plugin.py"
click node_events "https://github.com/smicallef/spiderfoot/blob/master/spiderfoot/event.py"
click node_pool "https://github.com/smicallef/spiderfoot/blob/master/spiderfoot/threadpool.py"
click node_discovery "https://github.com/smicallef/spiderfoot/tree/master/modules"
click node_threat "https://github.com/smicallef/spiderfoot/tree/master/modules"
click node_webdata "https://github.com/smicallef/spiderfoot/tree/master/modules"
click node_darkweb "https://github.com/smicallef/spiderfoot/tree/master/modules"
click node_cloud "https://github.com/smicallef/spiderfoot/tree/master/modules"
click node_storage "https://github.com/smicallef/spiderfoot/blob/master/spiderfoot/db.py"
click node_correlation "https://github.com/smicallef/spiderfoot/blob/master/spiderfoot/correlation.py"
click node_results "https://github.com/smicallef/spiderfoot/blob/master/spiderfoot/static/js/viz.js"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_web,node_cli,node_entry toneBlue
class node_scancontrol,node_common,node_plugin,node_events,node_pool toneAmber
class node_discovery,node_threat,node_webdata,node_darkweb,node_cloud toneMint
class node_storage,node_correlation,node_results toneRose
class node_user toneIndigo
```

## Components

### User interfaces

- [Web interface](https://github.com/smicallef/spiderfoot/blob/master/sfwebui.py): `sfwebui.py`
- [Command line](https://github.com/smicallef/spiderfoot/blob/master/sfcli.py): `sfcli.py`
- [Application entry](https://github.com/smicallef/spiderfoot/blob/master/sf.py): `sf.py`

### Scan execution

- [Scan control](https://github.com/smicallef/spiderfoot/blob/master/sfscan.py): `sfscan.py`
- [SpiderFoot services](https://github.com/smicallef/spiderfoot/blob/master/sflib.py): `sflib.py`
- [Plugin runtime](https://github.com/smicallef/spiderfoot/blob/master/spiderfoot/plugin.py): `spiderfoot/plugin.py`
- [Event model](https://github.com/smicallef/spiderfoot/blob/master/spiderfoot/event.py): `spiderfoot/event.py`
- [Thread pool](https://github.com/smicallef/spiderfoot/blob/master/spiderfoot/threadpool.py): `spiderfoot/threadpool.py`

### Discovery modules

- [Discovery and enrichment](https://github.com/smicallef/spiderfoot/tree/master/modules): `modules`
- [Threat intelligence](https://github.com/smicallef/spiderfoot/tree/master/modules): `modules`
- [Web and social discovery](https://github.com/smicallef/spiderfoot/tree/master/modules): `modules`
- [Dark-web search](https://github.com/smicallef/spiderfoot/tree/master/modules): `modules`
- [Cloud bucket checks](https://github.com/smicallef/spiderfoot/tree/master/modules): `modules`

### Analysis and results

- [Scan database](https://github.com/smicallef/spiderfoot/blob/master/spiderfoot/db.py): `spiderfoot/db.py`
- [Correlation engine](https://github.com/smicallef/spiderfoot/blob/master/spiderfoot/correlation.py): `spiderfoot/correlation.py`
- [Results and visualization](https://github.com/smicallef/spiderfoot/blob/master/spiderfoot/static/js/viz.js): `spiderfoot/static/js/viz.js`

### Other components

- **Analyst**

## Connections

- Analyst → Web interface: configures scan
- Analyst → Command line: runs scans
- Web interface → Scan control: starts scan (evidence: `sfwebui.py`)
- Application entry → Web interface: starts web UI (evidence: `sf.py`)
- Application entry → Scan control: starts scanning (evidence: `sf.py`)
- Scan control → SpiderFoot services: uses services (evidence: `sfscan.py`)
- Scan control → Plugin runtime: initializes plugins (evidence: `sfscan.py`)
- Plugin runtime → Thread pool: submits tasks (evidence: `spiderfoot/plugin.py`)
- Plugin runtime → Event model: publishes events (evidence: `spiderfoot/plugin.py`)
- Plugin runtime → Discovery and enrichment: dispatches events
- Plugin runtime → Threat intelligence: dispatches events
- Plugin runtime → Web and social discovery: dispatches events
- Plugin runtime → Dark-web search: dispatches events
- Plugin runtime → Cloud bucket checks: dispatches events
- Discovery and enrichment → Plugin runtime: publishes findings
- Threat intelligence → Plugin runtime: publishes findings
- Web and social discovery → Plugin runtime: publishes findings
- Dark-web search → Plugin runtime: publishes findings
- Cloud bucket checks → Plugin runtime: publishes findings
- Scan control → Scan database: stores scan data
- Correlation engine → Scan database: queries events (evidence: `spiderfoot/correlation.py`)
- Web interface → Scan database: reads scan data
- Web interface → Results and visualization: shows visualizations
