How qsert123/contact-scraper works
Data Scraper is a web application for finding supplier/dealer websites and extracting their contact details. The main workflow is browser query → API route → web search/filtering → parallel contact extraction → MongoDB persistence → results; separate history endpoints read saved searches and contacts. The route also builds stored documents using the model helpers. The graph reflects the sampled implementation; the HTML parser under data_extract/html_page/ appears standalone and is not wired into the documented API workflow.
Main entry points: scrape_direct.py Scrape and history routes · index.html Static UI · main.py FastAPI app · web_search.py Search and filtering
Components 11
Web interface
- Static UI
static/index.html - FastAPI app
api/main.py - Scrape and history routes
api/routes/scrape_direct.py
Search and extraction
- Search and filtering
collect_website/web_search.py - Contact extraction
data_extract/extract_js.py
History and storage
- Document builders
database/models.py - Database access
database/db.py
Other components
- User browser
- Web search service
- Supplier websites
- MongoDB
Connections 14
- User browser → Static UI: opens
- Static UI → FastAPI app: loads
- FastAPI app → Static UI: serves
- FastAPI app → Scrape and history routes: registers routes
- Static UI → Scrape and history routes: sends requests
- Scrape and history routes → Search and filtering: searches query
- Search and filtering → Web search service: queries
- Scrape and history routes → Contact extraction: extracts contacts
- Contact extraction → Supplier websites: fetches pages
- Scrape and history routes → Document builders: builds documents
- Scrape and history routes → Database access: gets database
- Database access → MongoDB: connects
- Scrape and history routes → MongoDB: writes results; reads history
- Scrape and history routes → Static UI: returns results