# rajib2017/darkdump architecture

> Architecture diagram of the GitHub repository rajib2017/darkdump, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rajib2017/darkdump
- Repository: https://github.com/rajib2017/darkdump

## Overview

##### Project Type & Purpose
- Darkdump is a Python-based CLI OSINT tool for deep-web investigations.
- It queries Ahmia.fi for .onion links, routes HTTP requests through Tor, scrapes pages for emails, metadata, keywords, images, etc., and presents/stores results locally.

##### File Structure Analysis
- darkdump.py: main entrypoint/CLI dispatcher.
- headers/agents.py: encapsulates HTTP user‐agent configuration.
- banner/banner.py: presentation layer for CLI UI.
- requirements.txt: lists Python dependencies (requests, stem, BeautifulSoup, Pillow, etc.).
- imgs/: sample output screenshots (documentation only).

##### Key README Insights
- External services: Ahmia API, Tor control port (9051), .onion sites via SOCKS5 proxy.
- Core features: search query, result count, optional scraping, image extraction, proxy toggle.
- Tor must be configured (ControlPort, HashedControlPassword).

a. Main Components
1. CLI Layer
- Argument parser (argparse in darkdump.py)
- Banner/Help display (banner/banner.py)
2. Query Module
- Ahmia API client (HTTP GET over standard network)
3. Scraper Module
- HTTP client configured with or without Tor proxy (headers/agents.py + requests/Stem)
- Tor control interface (authenticate, circuit management)
4. Parser/Extractor Subsystems
- Email & keyword extractor (regex/BeautifulSoup)
- Metadata extractor (HTTP headers, HTML meta tags)
- Image scraper (download, PIL processing)
- Social-media link finder
5. Storage/Output Module
- Console output & logging
- Local file storage for images and scraped data
6. External Dependencies
- Ahmia.fi HTTP API
- Tor daemon (SOCKS5 proxy at 127.0.0.1:9050; ControlPort 9051)
- Target .onion websites

b. Relationships & Data Flow
1. User invokes `darkdump.py` → CLI Layer
2. CLI Layer → Query Module → Ahmia API → returns onion URLs
3. For each URL, if “–scrape”:
- Scraper Module sends HTTP requests via Tor proxy → Target .onion site
- Parser modules extract data → Storage/Output
4. If “–images”: Scraper Module downloads images → Storage/Output

c. Architectural Patterns & Principles
- Layered architecture: CLI → Service → Scraping → Parsing → Storage
- Proxy pattern: HTTP client interchangeable between direct and Tor proxy modes
- Separation of concerns: distinct modules for networking, parsing, and UI

d. Key Technologies & Libraries
- Python 3.x, requests, stem (Tor control), BeautifulSoup4, Pillow, argparse

##### Essential Elements
- Clear component names matching code modules (darkdump.py, agents.py, banner.py).
- Data flows for both query-only and full-scrape modes.
- External dependencies as separate entities.
- Configuration points (Tor control port, proxy toggle).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI / UI Layer
    subgraph "CLI / UI Layer"
        DD["darkdump.py"]:::cli
        BN["banner/banner.py"]:::cli
    end

    %% Network & Proxy Layer
    subgraph "Network & Proxy Layer"
        QM["Query Module (Ahmia API)"]:::network
        SM["Scraper Module"]:::network
        AG["headers/agents.py"]:::network
    end

    %% Parsing / Extraction Layer
    subgraph "Parsing / Extraction Layer"
        PE["Email & Keyword Extractor"]:::parsing
        PM["Metadata Extractor"]:::parsing
        PI["Image Scraper"]:::parsing
        PS["Social Media Link Finder"]:::parsing
    end

    %% Storage / Output Layer
    subgraph "Storage / Output Layer"
        CO["Console Output & Logging"]:::storage
        FS["Local File Storage"]:::storage
    end

    %% External Services
    subgraph "External Services"
        AHM["Ahmia.fi API"]:::external
        TOR["Tor Daemon (SOCKS5 9050 / Control 9051)"]:::external
        ONIONS["Target .onion Sites"]:::external
    end

    %% Documentation & Assets
    subgraph "Documentation & Assets"
        REQ["requirements.txt"]:::docs
        IMG["imgs/"]:::docs
        RMD["README.md"]:::docs
        LIC["LICENSE"]:::docs
    end

    %% Connections
    DD -->|"invokes"| BN
    DD -->|"workflow control"| QM
    QM -->|"HTTPS"| AHM
    QM -->|"returns URLs"| SM
    SM -->|"uses HTTP client"| AG
    SM -->|"HTTP over SOCKS5:9050"| ONIONS
    SM -->|"Tor control:9051"| TOR
    SM -->|"passes HTML"| PE
    SM -->|"passes HTML"| PM
    SM -->|"downloads images"| PI
    SM -->|"passes HTML"| PS
    PE -->|"data"| CO
    PM -->|"data"| CO
    PS -->|"data"| CO
    PI -->|"images"| FS
    CO -->|"writes"| FS

    %% Click Events
    click DD "https://github.com/rajib2017/darkdump/blob/main/darkdump.py"
    click BN "https://github.com/rajib2017/darkdump/blob/main/banner/banner.py"
    click AG "https://github.com/rajib2017/darkdump/blob/main/headers/agents.py"
    click REQ "https://github.com/rajib2017/darkdump/blob/main/requirements.txt"
    click IMG "https://github.com/rajib2017/darkdump/tree/main/imgs/"
    click RMD "https://github.com/rajib2017/darkdump/blob/main/README.md"
    click LIC "https://github.com/rajib2017/darkdump/tree/main/LICENSE"

    %% Styles
    classDef cli fill:#DCEEFB,stroke:#0366d6
    classDef network fill:#D1F2D8,stroke:#27ae60
    classDef parsing fill:#FEF3C7,stroke:#f39c12
    classDef storage fill:#E5E7EB,stroke:#6B7280
    classDef external fill:#F0F0F0,stroke:#A0A0A0
    classDef docs fill:#F5E6EA,stroke:#A3425B
```
