# laramies/theharvester architecture

> Architecture diagram of the GitHub repository laramies/theharvester, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/laramies/theharvester
- Repository: https://github.com/laramies/theharvester

## Overview

##### Project Type and Purpose
- theHarvester is an open-source, Python-based OSINT reconnaissance tool.
- It functions primarily as a CLI application but also offers an optional RESTful API service (restfulHarvest) and can be containerized via Docker.
- Its core job is to orchestrate a set of discovery modules (to query external OSINT sources), parser modules (to normalize and filter raw results), and output/screenshot modules (to present findings).

##### High-Level Architecture & Major Components
a. Entry Points
- bin/theHarvester & theHarvester/__main__.py: CLI launcher
- bin/restfulHarvest & theHarvester/restfulHarvest.py: Flask-based HTTP server
b. Core Orchestrator
- theHarvester/lib/core.py: loads configuration & modules, coordinates the discovery-→-parse-→-output workflow
c. Configuration & Data
- theHarvester/data/*.yaml (API keys, proxies, wordlists)
- theHarvester/lib/api/static (static endpoints & examples)
d. Plugin Layers
1. Discovery Modules (theHarvester/discovery/*.py)
- Each wraps a public OSINT API (e.g., bingsearch.py → Bing, shodansearch.py → Shodan)
2. Parser Modules (theHarvester/parsers/*.py)
- Translate raw API responses into a unified internal model (hosts, emails, URLs)
3. Screenshot Module (theHarvester/screenshot/screenshot.py)
- Captures and stores snapshots of discovered subdomains
e. Output & Reporting
- Console (stdout formats: plain text, CSV, JSON)
- REST API JSON endpoints (via restfulHarvest)
- Screenshot files on disk
f. External Services
- A wide array of public OSINT APIs (Censys, VirusTotal, GitHub, etc.)
- DNS brute forcing using internal wordlists
g. Containerization & CI/CD
- Dockerfile & docker-compose.yml for building/running as a container
- GitHub Actions workflows for testing, linting, and publishing Docker images

##### Data Flow & Interactions
1. User invokes CLI or issues HTTP request to restfulHarvest.
2. Core Orchestrator reads configuration (API keys, proxies, wordlists).
3. Core dynamically loads chosen Discovery Modules.
4. Each Discovery Module issues HTTP requests to its target OSINT API, retrieves raw data.
5. Raw data is handed off to Parser Modules, which normalize and dedupe results.
6. Orchestrator aggregates parsed results, passes to Output Layer (console/JSON or screenshot).
7. For REST mode, orchestrator returns JSON payload; for CLI, writes to stdout or files.

##### Architectural Patterns & Principles
- Plugin Architecture: Discovery & Parser modules implement well-defined interfaces, enabling easy addition/removal.
- Layered Design: clear separation of concerns (Entry → Core → Plugins → Output).
- Single Responsibility: each module handles one OSINT source or one output format.
- Configuration-Driven: API keys, wordlists, and other parameters are externalized in YAML files.
- Container-First: designed to run in Docker for reproducible builds and deployments.

##### Essential Details to Capture
- Clear labels: “CLI Launcher,” “Flask REST Service,” “Core Orchestrator,”
“Discovery Plugins,” “Parser Plugins,” “Screenshot Engine,”
“Configuration Store,” “External OSINT APIs,” “Docker Container.”
- Data flows: “User → Core → Plugins → External APIs → Core → Output.”

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interface
    User("User Interface\n(CLI or HTTP Client)"):::external

    %% Docker Container Boundary
    subgraph "Docker Container" 
        direction TB
        CLI["CLI Launcher"]:::internal
        REST["Flask REST Service"]:::internal
        Core["Core Orchestrator"]:::internal
        Config[(Configuration Store)]:::config
        subgraph "Plugin Layers"
            direction TB
            Disc["Discovery Plugins"]:::internal
            Pars["Parser Plugins"]:::internal
            SS["Screenshot Engine"]:::internal
        end
        subgraph "Outputs"
            direction TB
            Console["Console Output"]:::internal
            JSON["REST JSON Output"]:::internal
            SSFiles[(Screenshot Files)]:::config
        end
    end

    %% External Services
    ExtOSINT["External OSINT APIs"]:::external

    %% CI/CD & Containerization
    CICD["CI/CD & Containerization"]:::infra

    %% Data Flow
    User -->|"CLI args"| CLI
    User -->|"HTTP request"| REST
    CLI -->|"invoke"| Core
    REST -->|"invoke"| Core
    Core -->|"load configs"| Config
    Core -->|"load modules"| Disc
    Disc -->|"HTTP/HTTPS"| ExtOSINT
    ExtOSINT -->|"raw data"| Pars
    Pars -->|"parsed results"| Core
    Core -->|"to console"| Console
    Core -->|"to JSON"| JSON
    Core -->|"capture"| SS
    SS -->|"store"| SSFiles
    Core -.->|"builds/pushes"| CICD

    %% Click Events
    click CLI "https://github.com/laramies/theharvester/tree/master/bin/theHarvester"
    click CLI "https://github.com/laramies/theharvester/blob/master/theHarvester/__main__.py"
    click REST "https://github.com/laramies/theharvester/tree/master/bin/restfulHarvest"
    click REST "https://github.com/laramies/theharvester/blob/master/theHarvester/restfulHarvest.py"
    click Core "https://github.com/laramies/theharvester/blob/master/theHarvester/lib/core.py"
    click Config "https://github.com/laramies/theharvester/blob/master/theHarvester/data/api-keys.yaml"
    click Config "https://github.com/laramies/theharvester/blob/master/theHarvester/data/proxies.yaml"
    click Config "https://github.com/laramies/theharvester/tree/master/theHarvester/data/wordlists/"
    click Config "https://github.com/laramies/theharvester/tree/master/theHarvester/lib/api/static/"
    click Disc "https://github.com/laramies/theharvester/tree/master/theHarvester/discovery/"
    click Pars "https://github.com/laramies/theharvester/tree/master/theHarvester/parsers/"
    click SS "https://github.com/laramies/theharvester/blob/master/theHarvester/screenshot/screenshot.py"
    click CICD "https://github.com/laramies/theharvester/tree/master/Dockerfile"
    click CICD "https://github.com/laramies/theharvester/blob/master/docker-compose.yml"
    click CICD "https://github.com/laramies/theharvester/blob/master/.github/workflows/docker-build-push.yml"
    click CICD "https://github.com/laramies/theharvester/blob/master/.github/workflows/dockerci.yml"
    click CICD "https://github.com/laramies/theharvester/blob/master/.github/workflows/theHarvester.yml"

    %% Styles
    classDef internal fill:#ccffcc,stroke:#333,stroke-width:1px;
    classDef config fill:#ffffcc,stroke:#333,stroke-width:1px;
    classDef external fill:#ccffff,stroke:#333,stroke-width:1px;
    classDef infra fill:#eeeeee,stroke:#333,stroke-width:1px;
```
