# nexvisionlab/url-and-website-safety-threat-intelligence-scanner architecture

> Architecture diagram of the GitHub repository nexvisionlab/url-and-website-safety-threat-intelligence-scanner, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner
- Repository: https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner

## Overview

url-safety-investigator is a Python CLI for assessing URL and website risk, with a specialized online-shop analyzer and shareable reports. The main workflow normalizes a target, checks cache, gathers local and live signals, aggregates them into an explainable verdict, and presents or stores the result. The shop workflow reuses URL investigation before adding shop-specific checks and a risk band. Edges follow the pipeline and analyzer excerpts, the SOURCE INDEX, and README; unsampled detector and shop internals are represented at subsystem level where their exact wiring is not shown.

Main entry points: [`usi/pipeline.py`](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/pipeline.py) (Investigation pipeline), [`usi/shop/analyzer.py`](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/shop/analyzer.py) (Shop analyzer), [`report.py`](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/report.py) (Report CLI), [`usi/cli.py`](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/cli.py) (Investigation CLI)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["User entry points"]
  node_cli["Investigation CLI<br/>[cli.py]"]
  node_shop_cli["Shop CLI<br/>[__main__.py]"]
  node_report_cli["Report CLI<br/>[report.py]"]
end

subgraph group_investigation["URL investigation"]
  node_pipeline["Investigation pipeline<br/>[pipeline.py]"]
  node_heuristics["Static URL heuristics"]
  node_registration["WHOIS and RDAP<br/>[whois_lookup.py]"]
  node_tls["TLS checks<br/>[tls_cert.py]"]
  node_cert_search["Certificate search<br/>[crtsh.py]"]
  node_page_fetch["Page fetching<br/>[fetcher.py]"]
  node_page_extract["Page extraction<br/>[extractor.py]"]
  node_content_checks["Content threat checks"]
  node_reputation["Reputation APIs"]
end

subgraph group_shop["Shop assessment"]
  node_shop_analyzer["Shop analyzer<br/>[analyzer.py]"]
  node_shop_pages["Shop page checks<br/>[pages.py]"]
  node_shop_fingerprint["Shop fingerprinting<br/>[fingerprint.py]"]
  node_shop_rules["Shop risk rules<br/>[rules.py]"]
  node_platforms["Marketplace guidance<br/>[platforms.py]"]
end

subgraph group_results["Results and state"]
  node_aggregator["Verdict aggregator<br/>[aggregator.py]"]
  node_cache[("Result cache<br/>[cache.py]")]
  node_formatter["Result formatter<br/>[formatter.py]"]
  node_report_builder["Report builder<br/>[report.py]"]
end

node_user(("User"))
node_external_sources(("Public data sources"))
node_reputation_services(("Reputation services"))
node_result["Verdict and findings"]
node_report_file["Markdown or HTML report"]

node_user -->|"submits URL"| node_cli
node_cli -->|"runs investigation"| node_pipeline
node_pipeline -->|"checks and stores"| node_cache
node_pipeline -->|"runs checks"| node_heuristics
node_pipeline -->|"looks up domain"| node_registration
node_registration -->|"queries registration data"| node_external_sources
node_pipeline -->|"checks certificate"| node_tls
node_pipeline -->|"checks lookalike certificates"| node_cert_search
node_pipeline -->|"fetches target"| node_page_fetch
node_page_fetch -->|"provides page content"| node_page_extract
node_pipeline -->|"checks extracted content"| node_content_checks
node_pipeline -->|"runs enabled checks"| node_reputation
node_reputation -->|"queries APIs"| node_reputation_services
node_pipeline -->|"aggregates signals"| node_aggregator
node_aggregator -->|"produces verdict"| node_result
node_pipeline -->|"formats result"| node_formatter
node_cli -->|"displays result"| node_formatter
node_user -->|"submits shop"| node_shop_cli
node_shop_cli -->|"runs assessment"| node_shop_analyzer
node_shop_analyzer -->|"recognizes marketplace"| node_platforms
node_shop_analyzer -->|"runs URL investigation"| node_pipeline
node_shop_analyzer -->|"checks shop pages"| node_shop_pages
node_shop_analyzer -->|"compares shop signals"| node_shop_fingerprint
node_shop_analyzer -->|"rates shop risk"| node_shop_rules
node_shop_analyzer -->|"returns shop assessment"| node_result
node_user -->|"requests report"| node_report_cli
node_report_cli -->|"investigates URLs"| node_pipeline
node_report_cli -->|"reads cached results"| node_cache
node_report_cli -->|"builds report"| node_report_builder
node_report_builder -->|"renders report"| node_report_file

click node_cli "https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/cli.py"
click node_shop_cli "https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/shop/__main__.py"
click node_report_cli "https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/report.py"
click node_pipeline "https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/pipeline.py"
click node_heuristics "https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/tree/main/usi/heuristics"
click node_registration "https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/lookups/whois_lookup.py"
click node_tls "https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/lookups/tls_cert.py"
click node_cert_search "https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/lookups/crtsh.py"
click node_page_fetch "https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/content/fetcher.py"
click node_page_extract "https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/content/extractor.py"
click node_content_checks "https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/tree/main/usi/content"
click node_reputation "https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/tree/main/usi/reputation"
click node_shop_analyzer "https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/shop/analyzer.py"
click node_shop_pages "https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/shop/pages.py"
click node_shop_fingerprint "https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/shop/fingerprint.py"
click node_shop_rules "https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/shop/rules.py"
click node_platforms "https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/shop/platforms.py"
click node_aggregator "https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/verdict/aggregator.py"
click node_cache "https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/cache.py"
click node_formatter "https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/output/formatter.py"
click node_report_builder "https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/output/report.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_shop_cli,node_report_cli,node_user toneBlue
class node_pipeline,node_heuristics,node_registration,node_tls,node_cert_search,node_page_fetch,node_page_extract,node_content_checks,node_reputation toneAmber
class node_shop_analyzer,node_shop_pages,node_shop_fingerprint,node_shop_rules,node_platforms toneMint
class node_aggregator,node_cache,node_formatter,node_report_builder toneRose
class node_external_sources,node_reputation_services,node_result,node_report_file toneIndigo
```

## Components

### User entry points

- [Investigation CLI](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/cli.py): `usi/cli.py`
- [Shop CLI](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/shop/__main__.py): `usi/shop/__main__.py`
- [Report CLI](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/report.py): `report.py`

### URL investigation

- [Investigation pipeline](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/pipeline.py): `usi/pipeline.py`
- [Static URL heuristics](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/tree/main/usi/heuristics): `usi/heuristics`
- [WHOIS and RDAP](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/lookups/whois_lookup.py): `usi/lookups/whois_lookup.py`
- [TLS checks](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/lookups/tls_cert.py): `usi/lookups/tls_cert.py`
- [Certificate search](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/lookups/crtsh.py): `usi/lookups/crtsh.py`
- [Page fetching](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/content/fetcher.py): `usi/content/fetcher.py`
- [Page extraction](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/content/extractor.py): `usi/content/extractor.py`
- [Content threat checks](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/tree/main/usi/content): `usi/content`
- [Reputation APIs](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/tree/main/usi/reputation): `usi/reputation`

### Shop assessment

- [Shop analyzer](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/shop/analyzer.py): `usi/shop/analyzer.py`
- [Shop page checks](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/shop/pages.py): `usi/shop/pages.py`
- [Shop fingerprinting](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/shop/fingerprint.py): `usi/shop/fingerprint.py`
- [Shop risk rules](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/shop/rules.py): `usi/shop/rules.py`
- [Marketplace guidance](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/shop/platforms.py): `usi/shop/platforms.py`

### Results and state

- [Verdict aggregator](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/verdict/aggregator.py): `usi/verdict/aggregator.py`
- [Result cache](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/cache.py): `usi/cache.py`
- [Result formatter](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/output/formatter.py): `usi/output/formatter.py`
- [Report builder](https://github.com/nexvisionlab/url-and-website-safety-threat-intelligence-scanner/blob/main/usi/output/report.py): `usi/output/report.py`

### Other components

- **User**
- **Public data sources**
- **Reputation services**
- **Verdict and findings**
- **Markdown or HTML report**

## Connections

- User → Investigation CLI: submits URL
- Investigation CLI → Investigation pipeline: runs investigation (evidence: `usi/cli.py`)
- Investigation pipeline → Result cache: checks and stores (evidence: `usi/pipeline.py`)
- Investigation pipeline → Static URL heuristics: runs checks (evidence: `usi/pipeline.py`)
- Investigation pipeline → WHOIS and RDAP: looks up domain (evidence: `usi/pipeline.py`)
- WHOIS and RDAP → Public data sources: queries registration data (evidence: `usi/lookups/whois_lookup.py`)
- Investigation pipeline → TLS checks: checks certificate (evidence: `usi/pipeline.py`)
- Investigation pipeline → Certificate search: checks lookalike certificates (evidence: `usi/pipeline.py`)
- Investigation pipeline → Page fetching: fetches target (evidence: `usi/pipeline.py`)
- Page fetching → Page extraction: provides page content (evidence: `usi/pipeline.py`)
- Investigation pipeline → Content threat checks: checks extracted content (evidence: `usi/pipeline.py`)
- Investigation pipeline → Reputation APIs: runs enabled checks (evidence: `usi/pipeline.py`)
- Reputation APIs → Reputation services: queries APIs (evidence: `usi/reputation/base.py`)
- Investigation pipeline → Verdict aggregator: aggregates signals (evidence: `usi/pipeline.py`)
- Verdict aggregator → Verdict and findings: produces verdict
- Investigation pipeline → Result formatter: formats result (evidence: `usi/pipeline.py`)
- Investigation CLI → Result formatter: displays result (evidence: `usi/cli.py`)
- User → Shop CLI: submits shop
- Shop CLI → Shop analyzer: runs assessment
- Shop analyzer → Marketplace guidance: recognizes marketplace (evidence: `usi/shop/analyzer.py`)
- Shop analyzer → Investigation pipeline: runs URL investigation (evidence: `usi/shop/analyzer.py`)
- Shop analyzer → Shop page checks: checks shop pages (evidence: `usi/shop/analyzer.py`)
- Shop analyzer → Shop fingerprinting: compares shop signals (evidence: `usi/shop/analyzer.py`)
- Shop analyzer → Shop risk rules: rates shop risk (evidence: `usi/shop/analyzer.py`)
- Shop analyzer → Verdict and findings: returns shop assessment
- User → Report CLI: requests report
- Report CLI → Investigation pipeline: investigates URLs (evidence: `report.py`)
- Report CLI → Result cache: reads cached results (evidence: `report.py`)
- Report CLI → Report builder: builds report (evidence: `report.py`)
- Report builder → Markdown or HTML report: renders report
