How eousuf/bd-bank_address_scraper works
This is a command-line scraper that turns bank and listing-page seeds into per-bank branch and sub-branch records, then merges the results into a refreshable dataset. The main runtime is concentrated in one Python file, so the internal stages below are responsibilities within that file, not separate modules. The README documents the major pipeline and fallback behavior; the sampled source is partial, so uncertain internal call wiring is omitted.
Main entry points: bank_branch_scraper.py Scraper runner · banks.csv Bank seeds · parts Bank part files · banks_branches.json Merged dataset
Components 18
Inputs and control
- Scraper runner
bank_branch_scraper.py - Bank seeds
banks.csv
Site discovery
- Website discovery
bank_branch_scraper.py - Listing discovery
bank_branch_scraper.py - Search engines
- Wikipedia
Scrape and extract
- Bank websites
- Page fetching
bank_branch_scraper.py - Firecrawl
- Page pagination
bank_branch_scraper.py - Record extraction
bank_branch_scraper.py - JSON and AJAX
bank_branch_scraper.py - HTML extraction
bank_branch_scraper.py - PDF extraction
bank_branch_scraper.py - Record dedupe
bank_branch_scraper.py
Outputs and state
- Bank part files
output/parts - Merged dataset
output/banks_branches.json
Other components
- CLI user
Connections 22
- CLI user → Scraper runner: runs
- Scraper runner → Bank seeds: reads
- Scraper runner → Website discovery: processes banks
- Website discovery → Bank seeds: reads hints
- Website discovery → Search engines: searches
- Website discovery → Wikipedia: falls back
- Scraper runner → Listing discovery: finds pages
- Listing discovery → Bank seeds: reads seeds
- Listing discovery → Bank websites: checks sitemaps and links
- Scraper runner → Page fetching: fetches pages
- Page fetching → Bank websites: requests pages
- Page fetching → Firecrawl: escalates failures
- Scraper runner → Page pagination: paginates listings
- Page pagination → Page fetching: fetches next pages
- Scraper runner → Record extraction: extracts records
- Record extraction → JSON and AJAX: parses feeds
- Record extraction → HTML extraction: parses HTML
- Record extraction → PDF extraction: parses PDFs
- Scraper runner → Record dedupe: deduplicates records
- Scraper runner → Bank part files: checkpoints results
- Scraper runner → Merged dataset: merges results
- Scraper runner → Bank part files: reads resume state