How rajib2017/darkdump works
Project Type & Purpose
• Darkdump is a Python-based CLI OSINT tool for deep-web investigations.
• It queries Ahmia.fi for .onion links, routes HTTP requests through Tor, scrapes pages for emails, metadata, keywords, images, etc., and presents/stores results locally.
File Structure Analysis
• darkdump.py: main entrypoint/CLI dispatcher.
• headers/agents.py: encapsulates HTTP user‐agent configuration.
• banner/banner.py: presentation layer for CLI UI.
• requirements.txt: lists Python dependencies (requests, stem, BeautifulSoup, Pillow, etc.).
• imgs/: sample output screenshots (documentation only).
Key README Insights
• External services: Ahmia API, Tor control port (9051), .onion sites via SOCKS5 proxy.
• Core features: search query, result count, optional scraping, image extraction, proxy toggle.
• Tor must be configured (ControlPort, HashedControlPassword).
a. Main Components
1. CLI Layer
• Argument parser (argparse in darkdump.py)
• Banner/Help display (banner/banner.py)
2. Query Module
• Ahmia API client (HTTP GET over standard network)
3. Scraper Module
• HTTP client configured with or without Tor proxy (headers/agents.py + requests/Stem)
• Tor control interface (authenticate, circuit management)
4. Parser/Extractor Subsystems
• Email & keyword extractor (regex/BeautifulSoup)
• Metadata extractor (HTTP headers, HTML meta tags)
• Image scraper (download, PIL processing)
• Social-media link finder
5. Storage/Output Module
• Console output & logging
• Local file storage for images and scraped data
6. External Dependencies
• Ahmia.fi HTTP API
• Tor daemon (SOCKS5 proxy at 127.0.0.1:9050; ControlPort 9051)
• Target .onion websites
b. Relationships & Data Flow
1. User invokes darkdump.py → CLI Layer
2. CLI Layer → Query Module → Ahmia API → returns onion URLs
3. For each URL, if “–scrape”:
• Scraper Module sends HTTP requests via Tor proxy → Target .onion site
• Parser modules extract data → Storage/Output
4. If “–images”: Scraper Module downloads images → Storage/Output
c. Architectural Patterns & Principles
• Layered architecture: CLI → Service → Scraping → Parsing → Storage
• Proxy pattern: HTTP client interchangeable between direct and Tor proxy modes
• Separation of concerns: distinct modules for networking, parsing, and UI
d. Key Technologies & Libraries
• Python 3.x, requests, stem (Tor control), BeautifulSoup4, Pillow, argparse
Essential Elements
• Clear component names matching code modules (darkdump.py, agents.py, banner.py).
• Data flows for both query-only and full-scrape modes.
• External dependencies as separate entities.
• Configuration points (Tor control port, proxy toggle).