# eousuf/bd-bank_address_scraper architecture

> Architecture diagram of the GitHub repository eousuf/bd-bank_address_scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/eousuf/bd-bank_address_scraper
- Repository: https://github.com/eousuf/bd-bank_address_scraper

## Overview

This is a command-line scraper that turns bank and listing-page seeds into per-bank branch and sub-branch records, then merges the results into a refreshable dataset. The main runtime is concentrated in one Python file, so the internal stages below are responsibilities within that file, not separate modules. The README documents the major pipeline and fallback behavior; the sampled source is partial, so uncertain internal call wiring is omitted.

Main entry points: [`bank_branch_scraper.py`](https://github.com/eousuf/bd-bank_address_scraper/blob/main/bank_branch_scraper.py) (Scraper runner), [`banks.csv`](https://github.com/eousuf/bd-bank_address_scraper/blob/main/banks.csv) (Bank seeds), [`output/parts`](https://github.com/eousuf/bd-bank_address_scraper/tree/main/output/parts) (Bank part files), [`output/banks_branches.json`](https://github.com/eousuf/bd-bank_address_scraper/blob/main/output/banks_branches.json) (Merged dataset)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inputs["Inputs and control"]
  node_runner["Scraper runner"]
  node_bank_list["Bank seeds<br/>[banks.csv]"]
end

subgraph group_discovery["Site discovery"]
  node_website_discovery["Website discovery"]
  node_listing_discovery["Listing discovery"]
  node_search_engines["Search engines"]
  node_wikipedia["Wikipedia"]
end

subgraph group_scraping["Scrape and extract"]
  node_bank_sites["Bank websites"]
  node_page_fetch["Page fetching"]
  node_firecrawl["Firecrawl"]
  node_pagination["Page pagination"]
  node_record_extract["Record extraction"]
  node_json_feed["JSON and AJAX"]
  node_html_extract["HTML extraction"]
  node_pdf_extract["PDF extraction"]
  node_dedupe["Record dedupe"]
end

subgraph group_persistence["Outputs and state"]
  node_part_files["Bank part files"]
  node_merged_file["Merged dataset"]
end

node_operator(("CLI user"))

node_operator -->|"runs"| node_runner
node_runner -->|"reads"| node_bank_list
node_runner -->|"processes banks"| node_website_discovery
node_website_discovery -->|"reads hints"| node_bank_list
node_website_discovery -->|"searches"| node_search_engines
node_website_discovery -.->|"falls back"| node_wikipedia
node_runner -->|"finds pages"| node_listing_discovery
node_listing_discovery -->|"reads seeds"| node_bank_list
node_listing_discovery -->|"checks sitemaps and links"| node_bank_sites
node_runner -->|"fetches pages"| node_page_fetch
node_page_fetch -->|"requests pages"| node_bank_sites
node_page_fetch -.->|"escalates failures"| node_firecrawl
node_runner -->|"paginates listings"| node_pagination
node_pagination -->|"fetches next pages"| node_page_fetch
node_runner -->|"extracts records"| node_record_extract
node_record_extract -->|"parses feeds"| node_json_feed
node_record_extract -->|"parses HTML"| node_html_extract
node_record_extract -->|"parses PDFs"| node_pdf_extract
node_runner -->|"deduplicates records"| node_dedupe
node_runner -->|"checkpoints results"| node_part_files
node_runner -->|"merges results"| node_merged_file
node_runner -->|"reads resume state"| node_part_files

click node_runner "https://github.com/eousuf/bd-bank_address_scraper/blob/main/bank_branch_scraper.py"
click node_bank_list "https://github.com/eousuf/bd-bank_address_scraper/blob/main/banks.csv"
click node_website_discovery "https://github.com/eousuf/bd-bank_address_scraper/blob/main/bank_branch_scraper.py"
click node_listing_discovery "https://github.com/eousuf/bd-bank_address_scraper/blob/main/bank_branch_scraper.py"
click node_page_fetch "https://github.com/eousuf/bd-bank_address_scraper/blob/main/bank_branch_scraper.py"
click node_pagination "https://github.com/eousuf/bd-bank_address_scraper/blob/main/bank_branch_scraper.py"
click node_record_extract "https://github.com/eousuf/bd-bank_address_scraper/blob/main/bank_branch_scraper.py"
click node_json_feed "https://github.com/eousuf/bd-bank_address_scraper/blob/main/bank_branch_scraper.py"
click node_html_extract "https://github.com/eousuf/bd-bank_address_scraper/blob/main/bank_branch_scraper.py"
click node_pdf_extract "https://github.com/eousuf/bd-bank_address_scraper/blob/main/bank_branch_scraper.py"
click node_dedupe "https://github.com/eousuf/bd-bank_address_scraper/blob/main/bank_branch_scraper.py"
click node_part_files "https://github.com/eousuf/bd-bank_address_scraper/tree/main/output/parts"
click node_merged_file "https://github.com/eousuf/bd-bank_address_scraper/blob/main/output/banks_branches.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runner,node_bank_list,node_operator toneBlue
class node_website_discovery,node_listing_discovery,node_search_engines,node_wikipedia toneAmber
class node_bank_sites,node_page_fetch,node_firecrawl,node_pagination,node_record_extract,node_json_feed,node_html_extract,node_pdf_extract,node_dedupe toneMint
class node_part_files,node_merged_file toneRose
```

## Components

### Inputs and control

- [Scraper runner](https://github.com/eousuf/bd-bank_address_scraper/blob/main/bank_branch_scraper.py): `bank_branch_scraper.py`
- [Bank seeds](https://github.com/eousuf/bd-bank_address_scraper/blob/main/banks.csv): `banks.csv`

### Site discovery

- [Website discovery](https://github.com/eousuf/bd-bank_address_scraper/blob/main/bank_branch_scraper.py): `bank_branch_scraper.py`
- [Listing discovery](https://github.com/eousuf/bd-bank_address_scraper/blob/main/bank_branch_scraper.py): `bank_branch_scraper.py`
- **Search engines**
- **Wikipedia**

### Scrape and extract

- **Bank websites**
- [Page fetching](https://github.com/eousuf/bd-bank_address_scraper/blob/main/bank_branch_scraper.py): `bank_branch_scraper.py`
- **Firecrawl**
- [Page pagination](https://github.com/eousuf/bd-bank_address_scraper/blob/main/bank_branch_scraper.py): `bank_branch_scraper.py`
- [Record extraction](https://github.com/eousuf/bd-bank_address_scraper/blob/main/bank_branch_scraper.py): `bank_branch_scraper.py`
- [JSON and AJAX](https://github.com/eousuf/bd-bank_address_scraper/blob/main/bank_branch_scraper.py): `bank_branch_scraper.py`
- [HTML extraction](https://github.com/eousuf/bd-bank_address_scraper/blob/main/bank_branch_scraper.py): `bank_branch_scraper.py`
- [PDF extraction](https://github.com/eousuf/bd-bank_address_scraper/blob/main/bank_branch_scraper.py): `bank_branch_scraper.py`
- [Record dedupe](https://github.com/eousuf/bd-bank_address_scraper/blob/main/bank_branch_scraper.py): `bank_branch_scraper.py`

### Outputs and state

- [Bank part files](https://github.com/eousuf/bd-bank_address_scraper/tree/main/output/parts): `output/parts`
- [Merged dataset](https://github.com/eousuf/bd-bank_address_scraper/blob/main/output/banks_branches.json): `output/banks_branches.json`

### Other components

- **CLI user**

## Connections

- CLI user → Scraper runner: runs
- Scraper runner → Bank seeds: reads
- Scraper runner → Website discovery: processes banks
- Website discovery → Bank seeds: reads hints
- Website discovery → Search engines: searches
- Website discovery → Wikipedia: falls back
- Scraper runner → Listing discovery: finds pages
- Listing discovery → Bank seeds: reads seeds
- Listing discovery → Bank websites: checks sitemaps and links
- Scraper runner → Page fetching: fetches pages
- Page fetching → Bank websites: requests pages
- Page fetching → Firecrawl: escalates failures
- Scraper runner → Page pagination: paginates listings
- Page pagination → Page fetching: fetches next pages
- Scraper runner → Record extraction: extracts records
- Record extraction → JSON and AJAX: parses feeds
- Record extraction → HTML extraction: parses HTML
- Record extraction → PDF extraction: parses PDFs
- Scraper runner → Record dedupe: deduplicates records
- Scraper runner → Bank part files: checkpoints results
- Scraper runner → Merged dataset: merges results
- Scraper runner → Bank part files: reads resume state