# gabih1/uk-vat-identifier-discovery architecture

> Architecture diagram of the GitHub repository gabih1/uk-vat-identifier-discovery, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gabih1/uk-vat-identifier-discovery
- Repository: https://github.com/gabih1/uk-vat-identifier-discovery

## Overview

This is a research proof of concept for discovering UK company VAT candidates from public sources, then checking candidates with HMRC and reviewing whether the returned legal entity matches. The principal workflow uses a Companies House sample, tries guessed or supplied domains, scans pages, extracts candidates, and records evidence; search, Common Crawl, marketplace, and procurement experiments are distinct discovery routes. Source excerpts show shared extraction and search calls, but do not show a production pipeline linking discovery to HMRC, so that documented handoff is dashed.

Main entry points: [`esantion_1000_companii.csv`](https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/esantion_1000_companii.csv) (Company sample), [`domain_guessing.py`](https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/domain_guessing.py) (Live-site scan), [`vat_utils.py`](https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/vat_utils.py) (VAT extraction), [`experiments/search_utils.py`](https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/experiments/search_utils.py) (Search scan runner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_sampling["Company sample"]
  node_sample_csv["Company sample"]
  node_sample_prep["Sample preparation<br/>[pregatire.py]"]
end

subgraph group_web["Web discovery"]
  node_live_scan["Live-site scan<br/>[domain_guessing.py]"]
end

subgraph group_sources["Alternative sources"]
  node_search_ddg["DuckDuckGo search"]
  node_search_google["Google search"]
  node_search_runner["Search scan runner<br/>[search_utils.py]"]
  node_common_crawl["Common Crawl scan"]
  node_amazon["Amazon seller scan"]
  node_contracts["Contracts Finder"]
end

subgraph group_analysis["Candidate analysis"]
  node_candidate_csv["Discovery results<br/>[openweb.csv]"]
  node_vat_utils["VAT extraction<br/>[vat_utils.py]"]
  node_discovery_utils["Discovery helpers<br/>[discovery_utils.py]"]
end

subgraph group_validation["Verification"]
  node_hmrc_client["HMRC VAT checker<br/>[hmrc_validate.py]"]
  node_hmrc{{"HMRC API"}}
  node_entity_review["Entity match review"]
end

node_companies_house(("Companies House"))

node_companies_house -->|"provides records"| node_sample_prep
node_sample_prep -->|"writes sample"| node_sample_csv
node_sample_csv -->|"supplies companies"| node_live_scan
node_live_scan -->|"uses helpers"| node_discovery_utils
node_live_scan -->|"extracts candidates"| node_vat_utils
node_live_scan -->|"writes results"| node_candidate_csv
node_sample_csv -->|"supplies companies"| node_search_ddg
node_search_ddg -->|"runs search"| node_search_runner
node_sample_csv -->|"supplies companies"| node_search_google
node_search_google -->|"runs search"| node_search_runner
node_search_runner -->|"scans results"| node_live_scan
node_search_runner -->|"uses helpers"| node_discovery_utils
node_sample_csv -->|"supplies companies"| node_common_crawl
node_common_crawl -->|"uses helpers"| node_discovery_utils
node_common_crawl -->|"extracts candidates"| node_vat_utils
node_sample_csv -.->|"research context"| node_amazon
node_amazon -->|"uses helpers"| node_discovery_utils
node_amazon -->|"extracts candidates"| node_vat_utils
node_contracts -->|"validates format"| node_vat_utils
node_vat_utils -.->|"provides candidate evidence"| node_entity_review
node_candidate_csv -.->|"candidate for checking"| node_hmrc_client
node_hmrc_client -->|"checks VAT"| node_hmrc
node_hmrc_client -->|"returns entity details"| node_entity_review

click node_sample_csv "https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/esantion_1000_companii.csv"
click node_sample_prep "https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/pregatire.py"
click node_live_scan "https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/domain_guessing.py"
click node_search_ddg "https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/experiments/duckduckgo_search_attempt.py"
click node_search_google "https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/experiments/google_search_attempt.py"
click node_search_runner "https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/experiments/search_utils.py"
click node_common_crawl "https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/common_crawl_check.py"
click node_amazon "https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/experiments/amazon_vat_pipeline.py"
click node_contracts "https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/experiments/contracts_finder_check.py"
click node_candidate_csv "https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/results/openweb.csv"
click node_vat_utils "https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/vat_utils.py"
click node_discovery_utils "https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/discovery_utils.py"
click node_hmrc_client "https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/hmrc_validate.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sample_csv,node_sample_prep toneBlue
class node_live_scan toneAmber
class node_search_ddg,node_search_google,node_search_runner,node_common_crawl,node_amazon,node_contracts toneMint
class node_candidate_csv,node_vat_utils,node_discovery_utils toneRose
class node_hmrc_client,node_hmrc,node_entity_review,node_companies_house toneIndigo
```

## Components

### Company sample

- [Company sample](https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/esantion_1000_companii.csv): `esantion_1000_companii.csv`
- [Sample preparation](https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/pregatire.py): `pregatire.py`

### Web discovery

- [Live-site scan](https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/domain_guessing.py): `domain_guessing.py`

### Alternative sources

- [DuckDuckGo search](https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/experiments/duckduckgo_search_attempt.py): `experiments/duckduckgo_search_attempt.py`
- [Google search](https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/experiments/google_search_attempt.py): `experiments/google_search_attempt.py`
- [Search scan runner](https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/experiments/search_utils.py): `experiments/search_utils.py`
- [Common Crawl scan](https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/common_crawl_check.py): `common_crawl_check.py`
- [Amazon seller scan](https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/experiments/amazon_vat_pipeline.py): `experiments/amazon_vat_pipeline.py`
- [Contracts Finder](https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/experiments/contracts_finder_check.py): `experiments/contracts_finder_check.py`

### Candidate analysis

- [Discovery results](https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/results/openweb.csv): `results/openweb.csv`
- [VAT extraction](https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/vat_utils.py): `vat_utils.py`
- [Discovery helpers](https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/discovery_utils.py): `discovery_utils.py`

### Verification

- [HMRC VAT checker](https://github.com/gabih1/uk-vat-identifier-discovery/blob/main/hmrc_validate.py): `hmrc_validate.py`
- **HMRC API**
- **Entity match review**

### Other components

- **Companies House**

## Connections

- Companies House → Sample preparation: provides records (evidence: `readme.md`)
- Sample preparation → Company sample: writes sample (evidence: `pregatire.py`)
- Company sample → Live-site scan: supplies companies (evidence: `domain_guessing.py`)
- Live-site scan → Discovery helpers: uses helpers (evidence: `domain_guessing.py`)
- Live-site scan → VAT extraction: extracts candidates (evidence: `domain_guessing.py`)
- Live-site scan → Discovery results: writes results (evidence: `domain_guessing.py`)
- Company sample → DuckDuckGo search: supplies companies (evidence: `experiments/duckduckgo_search_attempt.py`)
- DuckDuckGo search → Search scan runner: runs search (evidence: `experiments/duckduckgo_search_attempt.py`)
- Company sample → Google search: supplies companies (evidence: `experiments/google_search_attempt.py`)
- Google search → Search scan runner: runs search (evidence: `experiments/google_search_attempt.py`)
- Search scan runner → Live-site scan: scans results (evidence: `experiments/search_utils.py`)
- Search scan runner → Discovery helpers: uses helpers (evidence: `experiments/search_utils.py`)
- Company sample → Common Crawl scan: supplies companies (evidence: `common_crawl_check.py`)
- Common Crawl scan → Discovery helpers: uses helpers (evidence: `common_crawl_check.py`)
- Common Crawl scan → VAT extraction: extracts candidates (evidence: `common_crawl_check.py`)
- Company sample → Amazon seller scan: research context (evidence: `readme.md`)
- Amazon seller scan → Discovery helpers: uses helpers (evidence: `experiments/amazon_vat_pipeline.py`)
- Amazon seller scan → VAT extraction: extracts candidates (evidence: `experiments/amazon_vat_pipeline.py`)
- Contracts Finder → VAT extraction: validates format (evidence: `experiments/contracts_finder_check.py`)
- VAT extraction → Entity match review: provides candidate evidence
- Discovery results → HMRC VAT checker: candidate for checking (evidence: `readme.md`)
- HMRC VAT checker → HMRC API: checks VAT (evidence: `hmrc_validate.py`)
- HMRC VAT checker → Entity match review: returns entity details (evidence: `hmrc_validate.py`)
