How dirtyfilthy/freshonions-torscraper works
Fresh Onions is a Tor hidden-service crawler and discovery website. Its main workflow is to seed URLs, fetch onion pages through Tor, update domain/page records and extracted indicators, and expose searchable domain and page information through the web application. Separate capabilities include clone detection, port scanning, language detection, and optional Elasticsearch full-text search. The graph follows documented features and relationships visible in excerpts or the source index; unsampled subsystems are represented at their enclosing directory where appropriate, without inferring their internal call paths.
Main entry points: tor_scrapy.py Tor spider · domain.py Domain records · tor_elasticsearch.py Full-text index · app.py Web application
Components 21
Crawling and discovery
- Tor spider
torscraper/spiders/tor_scrapy.py - Crawler middleware
torscraper/middlewares.py - Scrape items
torscraper/items.py
Enrichment and analysis
- Indicator extraction
lib/helpers.py - Bitcoin validation
lib/bitcoin.py - Interesting paths
lib/interesting_paths.py - Language detection
lib/detect_language.py - Clone detection
lib/clone_detection.py - Port scanner
lib/portscanner.py - SSH fingerprinting
scripts/ssh_fingerprint.py - Email extraction
lib
Persistence and search
- Domain records
lib/tor_db/models/domain.py - Page records
lib/tor_db/models/page.py - MySQL database
lib/tor_db/db.py - Full-text index
lib/tor_elasticsearch.py
Web interface
- Web application
web/app.py - Search and views
lib/helpers.py - Response cache
lib/tor_cache.py
Other components
- Crawler operator
- URL sources
- Onion services
Connections 21
- Crawler operator → Tor spider: starts crawl
- URL sources → Tor spider: provides seeds
- Tor spider → Onion services: fetches pages
- Tor spider → Domain records: updates records
torscraper/spiders/tor_scrapy.py - Tor spider → Page records: updates records
torscraper/spiders/tor_scrapy.py - Tor spider → Bitcoin validation: validates addresses
torscraper/spiders/tor_scrapy.py - Tor spider → Indicator extraction: extracts indicators
- Domain records → MySQL database: persists domain data
lib/tor_db/models/domain.py - Page records → MySQL database: persists page data
lib/tor_db/models/page.py - Tor spider → Full-text index: indexes pages
torscraper/spiders/tor_scrapy.py - Domain records → Full-text index: indexes domains
lib/tor_db/models/domain.py - Domain records → Language detection: detects language
lib/tor_db/models/domain.py - Domain records → Interesting paths: finds useful paths
lib/tor_db/models/domain.py - Web application → Search and views: builds search results
web/app.py - Search and views → Full-text index: searches pages
lib/helpers.py - Web application → Domain records: serves domain data
web/app.py - Web application → Response cache: caches responses
web/app.py - Clone detection → Full-text index: uses index
lib/clone_detection.py - Port scanner → Domain records: records open ports
lib/portscanner.py - SSH fingerprinting → Domain records: records fingerprints
- Tor spider → Email extraction: extracts addresses
torscraper/spiders/tor_scrapy.py