# qsert123/contact-scraper architecture

> Architecture diagram of the GitHub repository qsert123/contact-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/qsert123/contact-scraper
- Repository: https://github.com/qsert123/contact-scraper

## Overview

Data Scraper is a web application for finding supplier/dealer websites and extracting their contact details. The main workflow is browser query → API route → web search/filtering → parallel contact extraction → MongoDB persistence → results; separate history endpoints read saved searches and contacts. The route also builds stored documents using the model helpers. The graph reflects the sampled implementation; the HTML parser under `data_extract/html_page/` appears standalone and is not wired into the documented API workflow.

Main entry points: [`api/routes/scrape_direct.py`](https://github.com/qsert123/contact-scraper/blob/main/api/routes/scrape_direct.py) (Scrape and history routes), [`static/index.html`](https://github.com/qsert123/contact-scraper/blob/main/static/index.html) (Static UI), [`api/main.py`](https://github.com/qsert123/contact-scraper/blob/main/api/main.py) (FastAPI app), [`collect_website/web_search.py`](https://github.com/qsert123/contact-scraper/blob/main/collect_website/web_search.py) (Search and filtering)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interface["Web interface"]
  node_ui["Static UI<br/>[index.html]"]
  node_app["FastAPI app<br/>[main.py]"]
  node_routes["Scrape and history routes<br/>[scrape_direct.py]"]
end

subgraph group_scraping["Search and extraction"]
  node_search["Search and filtering<br/>[web_search.py]"]
  node_extract["Contact extraction<br/>[extract_js.py]"]
end

subgraph group_storage["History and storage"]
  node_models["Document builders<br/>[models.py]"]
  node_db_access["Database access<br/>[db.py]"]
end

node_browser(("User browser"))
node_search_engine(("Web search service"))
node_websites(("Supplier websites"))
node_mongo[("MongoDB")]

node_browser -->|"opens"| node_ui
node_ui -->|"loads"| node_app
node_app -->|"serves"| node_ui
node_app -->|"registers routes"| node_routes
node_ui -->|"sends requests"| node_routes
node_routes -->|"searches query"| node_search
node_search -->|"queries"| node_search_engine
node_routes -->|"extracts contacts"| node_extract
node_extract -->|"fetches pages"| node_websites
node_routes -->|"builds documents"| node_models
node_routes -->|"gets database"| node_db_access
node_db_access -->|"connects"| node_mongo
node_routes -->|"writes results; reads history"| node_mongo
node_routes -->|"returns results"| node_ui

click node_ui "https://github.com/qsert123/contact-scraper/blob/main/static/index.html"
click node_app "https://github.com/qsert123/contact-scraper/blob/main/api/main.py"
click node_routes "https://github.com/qsert123/contact-scraper/blob/main/api/routes/scrape_direct.py"
click node_search "https://github.com/qsert123/contact-scraper/blob/main/collect_website/web_search.py"
click node_extract "https://github.com/qsert123/contact-scraper/blob/main/data_extract/extract_js.py"
click node_models "https://github.com/qsert123/contact-scraper/blob/main/database/models.py"
click node_db_access "https://github.com/qsert123/contact-scraper/blob/main/database/db.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ui,node_app,node_routes,node_browser toneBlue
class node_search,node_extract,node_mongo toneAmber
class node_models,node_db_access toneMint
class node_search_engine,node_websites toneIndigo
```

## Components

### Web interface

- [Static UI](https://github.com/qsert123/contact-scraper/blob/main/static/index.html): `static/index.html`
- [FastAPI app](https://github.com/qsert123/contact-scraper/blob/main/api/main.py): `api/main.py`
- [Scrape and history routes](https://github.com/qsert123/contact-scraper/blob/main/api/routes/scrape_direct.py): `api/routes/scrape_direct.py`

### Search and extraction

- [Search and filtering](https://github.com/qsert123/contact-scraper/blob/main/collect_website/web_search.py): `collect_website/web_search.py`
- [Contact extraction](https://github.com/qsert123/contact-scraper/blob/main/data_extract/extract_js.py): `data_extract/extract_js.py`

### History and storage

- [Document builders](https://github.com/qsert123/contact-scraper/blob/main/database/models.py): `database/models.py`
- [Database access](https://github.com/qsert123/contact-scraper/blob/main/database/db.py): `database/db.py`

### Other components

- **User browser**
- **Web search service**
- **Supplier websites**
- **MongoDB**

## Connections

- User browser → Static UI: opens
- Static UI → FastAPI app: loads
- FastAPI app → Static UI: serves
- FastAPI app → Scrape and history routes: registers routes
- Static UI → Scrape and history routes: sends requests
- Scrape and history routes → Search and filtering: searches query
- Search and filtering → Web search service: queries
- Scrape and history routes → Contact extraction: extracts contacts
- Contact extraction → Supplier websites: fetches pages
- Scrape and history routes → Document builders: builds documents
- Scrape and history routes → Database access: gets database
- Database access → MongoDB: connects
- Scrape and history routes → MongoDB: writes results; reads history
- Scrape and history routes → Static UI: returns results