# gorkemguler/metascout architecture

> Architecture diagram of the GitHub repository gorkemguler/metascout, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gorkemguler/metascout
- Repository: https://github.com/gorkemguler/metascout

## Overview

MetaScout discovers publicly exposed documents, downloads and analyzes them for metadata and optional content leaks, then produces reports. The main workflow is coordinated by the scan pipeline; users can initiate it through the CLI, local web UI, or job-based REST API. Discovery includes crawling, sitemaps, archive/search sources, subdomains, and critical-file checks. The README documents HTML, JSON, and optional PDF reports, plus scan history/diffs. Some discovery and pipeline details are unsampled, so only documented or indexed relationships are shown.

Main entry points: [`src/metascout/pipeline.py`](https://github.com/gorkemguler/metascout/blob/main/src/metascout/pipeline.py) (Scan pipeline), [`src/metascout/web.py`](https://github.com/gorkemguler/metascout/blob/main/src/metascout/web.py) (Local web UI), [`src/metascout/content_scan/__init__.py`](https://github.com/gorkemguler/metascout/blob/main/src/metascout/content_scan/__init__.py) (Content scanning), [`src/metascout/cli.py`](https://github.com/gorkemguler/metascout/blob/main/src/metascout/cli.py) (CLI)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["User interfaces"]
  node_cli["CLI<br/>[cli.py]"]
  node_web["Local web UI<br/>[web.py]"]
  node_api["REST API<br/>[app.py]"]
  node_jobs["Background jobs<br/>[jobs.py]"]
  node_config["Scan settings<br/>[config.py]"]
end

subgraph group_discovery["Discovery"]
  node_discovery_core["Crawl and sitemap<br/>[crawler.py]"]
  node_wayback["Wayback discovery<br/>[wayback.py]"]
  node_ddgs["DDGS search<br/>[ddgs_search.py]"]
  node_search_apis["Search APIs<br/>[search_engines.py]"]
  node_subdomains["Subdomain enumeration<br/>[subdomains.py]"]
end

subgraph group_analysis["Document analysis"]
  node_pipeline["Scan pipeline<br/>[pipeline.py]"]
  node_downloader["Document downloader<br/>[downloader.py]"]
  node_metadata_extract["Metadata extraction"]
  node_metadata_analyze["Metadata analysis<br/>[analyzer.py]"]
  node_content_scan["Content scanning<br/>[__init__.py]"]
  node_text_extract["Text and OCR<br/>[text_extract.py]"]
  node_pii["PII patterns<br/>[pii_patterns.py]"]
  node_visual_signature["Visual signatures"]
  node_models["Scan result models<br/>[models.py]"]
end

subgraph group_delivery["Results and reports"]
  node_reports["HTML and JSON reports<br/>[html_report.py]"]
  node_pdf["PDF report<br/>[pdf_report.py]"]
  node_diff["Report comparison<br/>[diff.py]"]
end

node_user(("Scanner"))
node_search_services(("Search providers"))
node_web_targets(("Target websites"))
node_exiftool(("ExifTool"))

node_user -->|"runs scan"| node_cli
node_user -->|"starts scan"| node_web
node_user -->|"submits job"| node_api
node_cli -->|"runs scan"| node_pipeline
node_web -->|"runs scan"| node_pipeline
node_api -->|"submits job"| node_jobs
node_jobs -->|"runs scan"| node_pipeline
node_pipeline -->|"uses settings"| node_config
node_pipeline -->|"discovers documents"| node_discovery_core
node_pipeline -->|"queries archive"| node_wayback
node_pipeline -->|"queries DDGS"| node_ddgs
node_pipeline -->|"queries providers"| node_search_apis
node_pipeline -->|"enumerates subdomains"| node_subdomains
node_pipeline -->|"downloads files"| node_downloader
node_downloader -->|"fetches documents"| node_web_targets
node_pipeline -->|"extracts metadata"| node_metadata_extract
node_metadata_extract -->|"invokes"| node_exiftool
node_pipeline -->|"analyzes metadata"| node_metadata_analyze
node_metadata_analyze -->|"checks internal hosts"| node_pii
node_pipeline -->|"scans content"| node_content_scan
node_content_scan -->|"extracts text"| node_text_extract
node_text_extract -->|"checks signatures"| node_visual_signature
node_content_scan -->|"detects patterns"| node_pii
node_pipeline -->|"builds findings"| node_models
node_pipeline -->|"renders reports"| node_reports
node_pipeline -->|"renders PDF"| node_pdf
node_web -->|"compares runs"| node_diff
node_search_apis -.->|"queries"| node_search_services

click node_cli "https://github.com/gorkemguler/metascout/blob/main/src/metascout/cli.py"
click node_web "https://github.com/gorkemguler/metascout/blob/main/src/metascout/web.py"
click node_api "https://github.com/gorkemguler/metascout/blob/main/src/metascout/api/app.py"
click node_jobs "https://github.com/gorkemguler/metascout/blob/main/src/metascout/api/jobs.py"
click node_config "https://github.com/gorkemguler/metascout/blob/main/src/metascout/config.py"
click node_pipeline "https://github.com/gorkemguler/metascout/blob/main/src/metascout/pipeline.py"
click node_discovery_core "https://github.com/gorkemguler/metascout/blob/main/src/metascout/discovery/crawler.py"
click node_wayback "https://github.com/gorkemguler/metascout/blob/main/src/metascout/discovery/wayback.py"
click node_ddgs "https://github.com/gorkemguler/metascout/blob/main/src/metascout/discovery/ddgs_search.py"
click node_search_apis "https://github.com/gorkemguler/metascout/blob/main/src/metascout/discovery/search_engines.py"
click node_subdomains "https://github.com/gorkemguler/metascout/blob/main/src/metascout/discovery/subdomains.py"
click node_downloader "https://github.com/gorkemguler/metascout/blob/main/src/metascout/downloader.py"
click node_metadata_extract "https://github.com/gorkemguler/metascout/blob/main/src/metascout/metadata/exiftool_wrapper.py"
click node_metadata_analyze "https://github.com/gorkemguler/metascout/blob/main/src/metascout/metadata/analyzer.py"
click node_content_scan "https://github.com/gorkemguler/metascout/blob/main/src/metascout/content_scan/__init__.py"
click node_text_extract "https://github.com/gorkemguler/metascout/blob/main/src/metascout/content_scan/text_extract.py"
click node_pii "https://github.com/gorkemguler/metascout/blob/main/src/metascout/content_scan/pii_patterns.py"
click node_visual_signature "https://github.com/gorkemguler/metascout/blob/main/src/metascout/content_scan/visual_signature.py"
click node_models "https://github.com/gorkemguler/metascout/blob/main/src/metascout/models.py"
click node_reports "https://github.com/gorkemguler/metascout/blob/main/src/metascout/report/html_report.py"
click node_pdf "https://github.com/gorkemguler/metascout/blob/main/src/metascout/report/pdf_report.py"
click node_diff "https://github.com/gorkemguler/metascout/blob/main/src/metascout/diff.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_web,node_api,node_jobs,node_config toneBlue
class node_discovery_core,node_wayback,node_ddgs,node_search_apis,node_subdomains toneAmber
class node_pipeline,node_downloader,node_metadata_extract,node_metadata_analyze,node_content_scan,node_text_extract,node_pii,node_visual_signature,node_models toneMint
class node_reports,node_pdf,node_diff toneRose
class node_user,node_search_services,node_web_targets,node_exiftool toneIndigo
```

## Components

### User interfaces

- [CLI](https://github.com/gorkemguler/metascout/blob/main/src/metascout/cli.py): `src/metascout/cli.py`
- [Local web UI](https://github.com/gorkemguler/metascout/blob/main/src/metascout/web.py): `src/metascout/web.py`
- [REST API](https://github.com/gorkemguler/metascout/blob/main/src/metascout/api/app.py): `src/metascout/api/app.py`
- [Background jobs](https://github.com/gorkemguler/metascout/blob/main/src/metascout/api/jobs.py): `src/metascout/api/jobs.py`
- [Scan settings](https://github.com/gorkemguler/metascout/blob/main/src/metascout/config.py): `src/metascout/config.py`

### Discovery

- [Crawl and sitemap](https://github.com/gorkemguler/metascout/blob/main/src/metascout/discovery/crawler.py): `src/metascout/discovery/crawler.py`
- [Wayback discovery](https://github.com/gorkemguler/metascout/blob/main/src/metascout/discovery/wayback.py): `src/metascout/discovery/wayback.py`
- [DDGS search](https://github.com/gorkemguler/metascout/blob/main/src/metascout/discovery/ddgs_search.py): `src/metascout/discovery/ddgs_search.py`
- [Search APIs](https://github.com/gorkemguler/metascout/blob/main/src/metascout/discovery/search_engines.py): `src/metascout/discovery/search_engines.py`
- [Subdomain enumeration](https://github.com/gorkemguler/metascout/blob/main/src/metascout/discovery/subdomains.py): `src/metascout/discovery/subdomains.py`

### Document analysis

- [Scan pipeline](https://github.com/gorkemguler/metascout/blob/main/src/metascout/pipeline.py): `src/metascout/pipeline.py`
- [Document downloader](https://github.com/gorkemguler/metascout/blob/main/src/metascout/downloader.py): `src/metascout/downloader.py`
- [Metadata extraction](https://github.com/gorkemguler/metascout/blob/main/src/metascout/metadata/exiftool_wrapper.py): `src/metascout/metadata/exiftool_wrapper.py`
- [Metadata analysis](https://github.com/gorkemguler/metascout/blob/main/src/metascout/metadata/analyzer.py): `src/metascout/metadata/analyzer.py`
- [Content scanning](https://github.com/gorkemguler/metascout/blob/main/src/metascout/content_scan/__init__.py): `src/metascout/content_scan/__init__.py`
- [Text and OCR](https://github.com/gorkemguler/metascout/blob/main/src/metascout/content_scan/text_extract.py): `src/metascout/content_scan/text_extract.py`
- [PII patterns](https://github.com/gorkemguler/metascout/blob/main/src/metascout/content_scan/pii_patterns.py): `src/metascout/content_scan/pii_patterns.py`
- [Visual signatures](https://github.com/gorkemguler/metascout/blob/main/src/metascout/content_scan/visual_signature.py): `src/metascout/content_scan/visual_signature.py`
- [Scan result models](https://github.com/gorkemguler/metascout/blob/main/src/metascout/models.py): `src/metascout/models.py`

### Results and reports

- [HTML and JSON reports](https://github.com/gorkemguler/metascout/blob/main/src/metascout/report/html_report.py): `src/metascout/report/html_report.py`
- [PDF report](https://github.com/gorkemguler/metascout/blob/main/src/metascout/report/pdf_report.py): `src/metascout/report/pdf_report.py`
- [Report comparison](https://github.com/gorkemguler/metascout/blob/main/src/metascout/diff.py): `src/metascout/diff.py`

### Other components

- **Scanner**
- **Search providers**
- **Target websites**
- **ExifTool**

## Connections

- Scanner → CLI: runs scan (evidence: `README.md`)
- Scanner → Local web UI: starts scan (evidence: `README.md`)
- Scanner → REST API: submits job (evidence: `src/metascout/api/app.py`)
- CLI → Scan pipeline: runs scan (evidence: `src/metascout/cli.py`)
- Local web UI → Scan pipeline: runs scan (evidence: `src/metascout/web.py`)
- REST API → Background jobs: submits job (evidence: `src/metascout/api/app.py`)
- Background jobs → Scan pipeline: runs scan (evidence: `src/metascout/api/app.py`)
- Scan pipeline → Scan settings: uses settings (evidence: `src/metascout/pipeline.py`)
- Scan pipeline → Crawl and sitemap: discovers documents (evidence: `src/metascout/pipeline.py`)
- Scan pipeline → Wayback discovery: queries archive (evidence: `README.md`)
- Scan pipeline → DDGS search: queries DDGS (evidence: `src/metascout/pipeline.py`)
- Scan pipeline → Search APIs: queries providers (evidence: `src/metascout/pipeline.py`)
- Scan pipeline → Subdomain enumeration: enumerates subdomains (evidence: `README.md`)
- Scan pipeline → Document downloader: downloads files (evidence: `src/metascout/pipeline.py`)
- Document downloader → Target websites: fetches documents (evidence: `README.md`)
- Scan pipeline → Metadata extraction: extracts metadata (evidence: `src/metascout/pipeline.py`)
- Metadata extraction → ExifTool: invokes (evidence: `src/metascout/metadata/exiftool_wrapper.py`)
- Scan pipeline → Metadata analysis: analyzes metadata (evidence: `src/metascout/pipeline.py`)
- Metadata analysis → PII patterns: checks internal hosts (evidence: `src/metascout/metadata/analyzer.py`)
- Scan pipeline → Content scanning: scans content (evidence: `src/metascout/pipeline.py`)
- Content scanning → Text and OCR: extracts text (evidence: `src/metascout/content_scan/__init__.py`)
- Text and OCR → Visual signatures: checks signatures (evidence: `src/metascout/content_scan/__init__.py`)
- Content scanning → PII patterns: detects patterns (evidence: `src/metascout/content_scan/__init__.py`)
- Scan pipeline → Scan result models: builds findings (evidence: `src/metascout/pipeline.py`)
- Scan pipeline → HTML and JSON reports: renders reports (evidence: `README.md`)
- Scan pipeline → PDF report: renders PDF (evidence: `README.md`)
- Local web UI → Report comparison: compares runs (evidence: `src/metascout/web.py`)
- Search APIs → Search providers: queries
