How laramies/theharvester works
Project Type and Purpose
• theHarvester is an open-source, Python-based OSINT reconnaissance tool.
• It functions primarily as a CLI application but also offers an optional RESTful API service (restfulHarvest) and can be containerized via Docker.
• Its core job is to orchestrate a set of discovery modules (to query external OSINT sources), parser modules (to normalize and filter raw results), and output/screenshot modules (to present findings).
High-Level Architecture & Major Components
a. Entry Points
• bin/theHarvester & theHarvester/__main__.py: CLI launcher
• bin/restfulHarvest & theHarvester/restfulHarvest.py: Flask-based HTTP server
b. Core Orchestrator
• theHarvester/lib/core.py: loads configuration & modules, coordinates the discovery-→-parse-→-output workflow
c. Configuration & Data
• theHarvester/data/*.yaml (API keys, proxies, wordlists)
• theHarvester/lib/api/static (static endpoints & examples)
d. Plugin Layers
1. Discovery Modules (theHarvester/discovery/*.py)
• Each wraps a public OSINT API (e.g., bingsearch.py → Bing, shodansearch.py → Shodan)
2. Parser Modules (theHarvester/parsers/*.py)
• Translate raw API responses into a unified internal model (hosts, emails, URLs)
3. Screenshot Module (theHarvester/screenshot/screenshot.py)
• Captures and stores snapshots of discovered subdomains
e. Output & Reporting
• Console (stdout formats: plain text, CSV, JSON)
• REST API JSON endpoints (via restfulHarvest)
• Screenshot files on disk
f. External Services
• A wide array of public OSINT APIs (Censys, VirusTotal, GitHub, etc.)
• DNS brute forcing using internal wordlists
g. Containerization & CI/CD
• Dockerfile & docker-compose.yml for building/running as a container
• GitHub Actions workflows for testing, linting, and publishing Docker images
Data Flow & Interactions
1. User invokes CLI or issues HTTP request to restfulHarvest.
2. Core Orchestrator reads configuration (API keys, proxies, wordlists).
3. Core dynamically loads chosen Discovery Modules.
4. Each Discovery Module issues HTTP requests to its target OSINT API, retrieves raw data.
5. Raw data is handed off to Parser Modules, which normalize and dedupe results.
6. Orchestrator aggregates parsed results, passes to Output Layer (console/JSON or screenshot).
7. For REST mode, orchestrator returns JSON payload; for CLI, writes to stdout or files.
Architectural Patterns & Principles
• Plugin Architecture: Discovery & Parser modules implement well-defined interfaces, enabling easy addition/removal.
• Layered Design: clear separation of concerns (Entry → Core → Plugins → Output).
• Single Responsibility: each module handles one OSINT source or one output format.
• Configuration-Driven: API keys, wordlists, and other parameters are externalized in YAML files.
• Container-First: designed to run in Docker for reproducible builds and deployments.
Essential Details to Capture
• Clear labels: “CLI Launcher,” “Flask REST Service,” “Core Orchestrator,”
“Discovery Plugins,” “Parser Plugins,” “Screenshot Engine,”
“Configuration Store,” “External OSINT APIs,” “Docker Container.”
• Data flows: “User → Core → Plugins → External APIs → Core → Output.”