# dirtyfilthy/freshonions-torscraper architecture

> Architecture diagram of the GitHub repository dirtyfilthy/freshonions-torscraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dirtyfilthy/freshonions-torscraper
- Repository: https://github.com/dirtyfilthy/freshonions-torscraper

## Overview

Fresh Onions is a Tor hidden-service crawler and discovery website. Its main workflow is to seed URLs, fetch onion pages through Tor, update domain/page records and extracted indicators, and expose searchable domain and page information through the web application. Separate capabilities include clone detection, port scanning, language detection, and optional Elasticsearch full-text search. The graph follows documented features and relationships visible in excerpts or the source index; unsampled subsystems are represented at their enclosing directory where appropriate, without inferring their internal call paths.

Main entry points: [`torscraper/spiders/tor_scrapy.py`](https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/torscraper/spiders/tor_scrapy.py) (Tor spider), [`lib/tor_db/models/domain.py`](https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/tor_db/models/domain.py) (Domain records), [`lib/tor_elasticsearch.py`](https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/tor_elasticsearch.py) (Full-text index), [`web/app.py`](https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/web/app.py) (Web application)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_crawl["Crawling and discovery"]
  node_spider["Tor spider<br/>[tor_scrapy.py]"]
  node_middleware["Crawler middleware<br/>[middlewares.py]"]
  node_items["Scrape items<br/>[items.py]"]
end

subgraph group_analysis["Enrichment and analysis"]
  node_indicators["Indicator extraction<br/>[helpers.py]"]
  node_bitcoin["Bitcoin validation<br/>[bitcoin.py]"]
  node_paths["Interesting paths"]
  node_language["Language detection<br/>[detect_language.py]"]
  node_clones["Clone detection<br/>[clone_detection.py]"]
  node_scanner["Port scanner<br/>[portscanner.py]"]
  node_ssh["SSH fingerprinting<br/>[ssh_fingerprint.py]"]
  node_email["Email extraction"]
end

subgraph group_storage["Persistence and search"]
  node_domain["Domain records<br/>[domain.py]"]
  node_page["Page records<br/>[page.py]"]
  node_mysql[("MySQL database<br/>[db.py]")]
  node_es[("Full-text index")]
end

subgraph group_web["Web interface"]
  node_webapp["Web application<br/>[app.py]"]
  node_webhelpers["Search and views<br/>[helpers.py]"]
  node_cache["Response cache<br/>[tor_cache.py]"]
end

node_operator(("Crawler operator"))
node_sources(("URL sources"))
node_onion(("Onion services"))

node_operator -.->|"starts crawl"| node_spider
node_sources -.->|"provides seeds"| node_spider
node_spider -.->|"fetches pages"| node_onion
node_spider -->|"updates records"| node_domain
node_spider -->|"updates records"| node_page
node_spider -->|"validates addresses"| node_bitcoin
node_spider -.->|"extracts indicators"| node_indicators
node_domain -->|"persists domain data"| node_mysql
node_page -->|"persists page data"| node_mysql
node_spider -->|"indexes pages"| node_es
node_domain -->|"indexes domains"| node_es
node_domain -->|"detects language"| node_language
node_domain -->|"finds useful paths"| node_paths
node_webapp -->|"builds search results"| node_webhelpers
node_webhelpers -->|"searches pages"| node_es
node_webapp -->|"serves domain data"| node_domain
node_webapp -->|"caches responses"| node_cache
node_clones -->|"uses index"| node_es
node_scanner -->|"records open ports"| node_domain
node_ssh -.->|"records fingerprints"| node_domain
node_spider -->|"extracts addresses"| node_email

click node_spider "https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/torscraper/spiders/tor_scrapy.py"
click node_middleware "https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/torscraper/middlewares.py"
click node_items "https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/torscraper/items.py"
click node_domain "https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/tor_db/models/domain.py"
click node_page "https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/tor_db/models/page.py"
click node_mysql "https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/tor_db/db.py"
click node_indicators "https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/helpers.py"
click node_bitcoin "https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/bitcoin.py"
click node_paths "https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/interesting_paths.py"
click node_language "https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/detect_language.py"
click node_clones "https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/clone_detection.py"
click node_scanner "https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/portscanner.py"
click node_es "https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/tor_elasticsearch.py"
click node_webapp "https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/web/app.py"
click node_webhelpers "https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/helpers.py"
click node_cache "https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/tor_cache.py"
click node_ssh "https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/scripts/ssh_fingerprint.py"
click node_email "https://github.com/dirtyfilthy/freshonions-torscraper/tree/master/lib"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_spider,node_middleware,node_items toneBlue
class node_indicators,node_bitcoin,node_paths,node_language,node_clones,node_scanner,node_ssh,node_email toneAmber
class node_domain,node_page,node_mysql,node_es toneMint
class node_webapp,node_webhelpers,node_cache toneRose
class node_operator,node_sources,node_onion toneIndigo
```

## Components

### Crawling and discovery

- [Tor spider](https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/torscraper/spiders/tor_scrapy.py): `torscraper/spiders/tor_scrapy.py`
- [Crawler middleware](https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/torscraper/middlewares.py): `torscraper/middlewares.py`
- [Scrape items](https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/torscraper/items.py): `torscraper/items.py`

### Enrichment and analysis

- [Indicator extraction](https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/helpers.py): `lib/helpers.py`
- [Bitcoin validation](https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/bitcoin.py): `lib/bitcoin.py`
- [Interesting paths](https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/interesting_paths.py): `lib/interesting_paths.py`
- [Language detection](https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/detect_language.py): `lib/detect_language.py`
- [Clone detection](https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/clone_detection.py): `lib/clone_detection.py`
- [Port scanner](https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/portscanner.py): `lib/portscanner.py`
- [SSH fingerprinting](https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/scripts/ssh_fingerprint.py): `scripts/ssh_fingerprint.py`
- [Email extraction](https://github.com/dirtyfilthy/freshonions-torscraper/tree/master/lib): `lib`

### Persistence and search

- [Domain records](https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/tor_db/models/domain.py): `lib/tor_db/models/domain.py`
- [Page records](https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/tor_db/models/page.py): `lib/tor_db/models/page.py`
- [MySQL database](https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/tor_db/db.py): `lib/tor_db/db.py`
- [Full-text index](https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/tor_elasticsearch.py): `lib/tor_elasticsearch.py`

### Web interface

- [Web application](https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/web/app.py): `web/app.py`
- [Search and views](https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/helpers.py): `lib/helpers.py`
- [Response cache](https://github.com/dirtyfilthy/freshonions-torscraper/blob/master/lib/tor_cache.py): `lib/tor_cache.py`

### Other components

- **Crawler operator**
- **URL sources**
- **Onion services**

## Connections

- Crawler operator → Tor spider: starts crawl
- URL sources → Tor spider: provides seeds
- Tor spider → Onion services: fetches pages
- Tor spider → Domain records: updates records (evidence: `torscraper/spiders/tor_scrapy.py`)
- Tor spider → Page records: updates records (evidence: `torscraper/spiders/tor_scrapy.py`)
- Tor spider → Bitcoin validation: validates addresses (evidence: `torscraper/spiders/tor_scrapy.py`)
- Tor spider → Indicator extraction: extracts indicators
- Domain records → MySQL database: persists domain data (evidence: `lib/tor_db/models/domain.py`)
- Page records → MySQL database: persists page data (evidence: `lib/tor_db/models/page.py`)
- Tor spider → Full-text index: indexes pages (evidence: `torscraper/spiders/tor_scrapy.py`)
- Domain records → Full-text index: indexes domains (evidence: `lib/tor_db/models/domain.py`)
- Domain records → Language detection: detects language (evidence: `lib/tor_db/models/domain.py`)
- Domain records → Interesting paths: finds useful paths (evidence: `lib/tor_db/models/domain.py`)
- Web application → Search and views: builds search results (evidence: `web/app.py`)
- Search and views → Full-text index: searches pages (evidence: `lib/helpers.py`)
- Web application → Domain records: serves domain data (evidence: `web/app.py`)
- Web application → Response cache: caches responses (evidence: `web/app.py`)
- Clone detection → Full-text index: uses index (evidence: `lib/clone_detection.py`)
- Port scanner → Domain records: records open ports (evidence: `lib/portscanner.py`)
- SSH fingerprinting → Domain records: records fingerprints
- Tor spider → Email extraction: extracts addresses (evidence: `torscraper/spiders/tor_scrapy.py`)