# nyn-05/lead_scraper architecture

> Architecture diagram of the GitHub repository nyn-05/lead_scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nyn-05/lead_scraper
- Repository: https://github.com/nyn-05/lead_scraper

## Overview

LEAD_SCRAPER is a Python toolkit for discovering Google Maps businesses, public LinkedIn/Instagram profiles, and photography-service prospects in public posts. Users initiate collection through the CLI or Streamlit dashboard; collection results are processed, persisted in SQLite, and exported as CSV, JSON, or Excel. The graph separates the three discovery workflows from shared configuration, processing, and storage. Some subsystem links rely on README descriptions because the relevant implementations were not fully excerpted.

Main entry points: [`main.py`](https://github.com/nyn-05/lead_scraper/blob/main/main.py) (CLI dispatch), [`lead_scraper/storage.py`](https://github.com/nyn-05/lead_scraper/blob/main/lead_scraper/storage.py) (Database and exports), [`lead_scraper/dashboard.py`](https://github.com/nyn-05/lead_scraper/blob/main/lead_scraper/dashboard.py) (Streamlit dashboard), [`lead_scraper/scrapers.py`](https://github.com/nyn-05/lead_scraper/blob/main/lead_scraper/scrapers.py) (Maps listings)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["User interfaces"]
  node_cli["CLI dispatch<br/>[main.py]"]
  node_dashboard["Streamlit dashboard<br/>[dashboard.py]"]
end

subgraph group_discovery["Lead discovery"]
  node_maps["Maps listings<br/>[scrapers.py]"]
  node_social["Social profiles<br/>[scrapers.py]"]
  node_photo["Photography collectors<br/>[scrapers.py]"]
end

subgraph group_processing["Processing and configuration"]
  node_settings["Configuration<br/>[config.py]"]
  node_normalize["Post normalization<br/>[processing.py]"]
  node_classify["Lead classification<br/>[processing.py]"]
  node_score["Lead scoring<br/>[processing.py]"]
end

subgraph group_persistence["Persistence and exports"]
  node_database_api["Database and exports<br/>[storage.py]"]
  node_sqlite[("SQLite database<br/>[leads.db]")]
  node_exports["CSV JSON Excel<br/>[storage.py]"]
end

node_user(("User"))
node_public_sources{{"Public sources"}}
node_search_providers{{"Search providers"}}
node_maps_service{{"Google Maps"}}

node_user -->|"runs command"| node_cli
node_user -->|"uses dashboard"| node_dashboard
node_cli -->|"dispatches Maps"| node_maps
node_cli -->|"dispatches social"| node_social
node_cli -->|"runs collectors"| node_photo
node_dashboard -->|"starts collection"| node_maps
node_dashboard -->|"starts collection"| node_social
node_dashboard -->|"starts collection"| node_photo
node_maps -->|"collects listings"| node_maps_service
node_social -->|"searches profiles"| node_search_providers
node_photo -->|"collects posts"| node_public_sources
node_cli -->|"loads settings"| node_settings
node_cli -->|"normalizes posts"| node_normalize
node_cli -->|"classifies posts"| node_classify
node_cli -->|"scores leads"| node_score
node_photo -->|"submits posts"| node_normalize
node_normalize -->|"feeds normalized posts"| node_classify
node_classify -->|"provides classification"| node_score
node_cli -->|"persists results"| node_database_api
node_maps -->|"stores listings"| node_database_api
node_social -->|"stores profiles"| node_database_api
node_dashboard -->|"reads stored data"| node_database_api
node_database_api -->|"reads and writes"| node_sqlite
node_database_api -->|"writes exports"| node_exports

click node_cli "https://github.com/nyn-05/lead_scraper/blob/main/main.py"
click node_dashboard "https://github.com/nyn-05/lead_scraper/blob/main/lead_scraper/dashboard.py"
click node_maps "https://github.com/nyn-05/lead_scraper/blob/main/lead_scraper/scrapers.py"
click node_social "https://github.com/nyn-05/lead_scraper/blob/main/lead_scraper/scrapers.py"
click node_photo "https://github.com/nyn-05/lead_scraper/blob/main/lead_scraper/scrapers.py"
click node_settings "https://github.com/nyn-05/lead_scraper/blob/main/lead_scraper/config.py"
click node_normalize "https://github.com/nyn-05/lead_scraper/blob/main/lead_scraper/processing.py"
click node_classify "https://github.com/nyn-05/lead_scraper/blob/main/lead_scraper/processing.py"
click node_score "https://github.com/nyn-05/lead_scraper/blob/main/lead_scraper/processing.py"
click node_database_api "https://github.com/nyn-05/lead_scraper/blob/main/lead_scraper/storage.py"
click node_sqlite "https://github.com/nyn-05/lead_scraper/blob/main/data/leads.db"
click node_exports "https://github.com/nyn-05/lead_scraper/blob/main/lead_scraper/storage.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_dashboard,node_user toneBlue
class node_maps,node_social,node_photo toneAmber
class node_settings,node_normalize,node_classify,node_score toneMint
class node_database_api,node_sqlite,node_exports toneRose
class node_public_sources,node_search_providers,node_maps_service toneIndigo
```

## Components

### User interfaces

- [CLI dispatch](https://github.com/nyn-05/lead_scraper/blob/main/main.py): `main.py`
- [Streamlit dashboard](https://github.com/nyn-05/lead_scraper/blob/main/lead_scraper/dashboard.py): `lead_scraper/dashboard.py`

### Lead discovery

- [Maps listings](https://github.com/nyn-05/lead_scraper/blob/main/lead_scraper/scrapers.py): `lead_scraper/scrapers.py`
- [Social profiles](https://github.com/nyn-05/lead_scraper/blob/main/lead_scraper/scrapers.py): `lead_scraper/scrapers.py`
- [Photography collectors](https://github.com/nyn-05/lead_scraper/blob/main/lead_scraper/scrapers.py): `lead_scraper/scrapers.py`

### Processing and configuration

- [Configuration](https://github.com/nyn-05/lead_scraper/blob/main/lead_scraper/config.py): `lead_scraper/config.py`
- [Post normalization](https://github.com/nyn-05/lead_scraper/blob/main/lead_scraper/processing.py): `lead_scraper/processing.py`
- [Lead classification](https://github.com/nyn-05/lead_scraper/blob/main/lead_scraper/processing.py): `lead_scraper/processing.py`
- [Lead scoring](https://github.com/nyn-05/lead_scraper/blob/main/lead_scraper/processing.py): `lead_scraper/processing.py`

### Persistence and exports

- [Database and exports](https://github.com/nyn-05/lead_scraper/blob/main/lead_scraper/storage.py): `lead_scraper/storage.py`
- [SQLite database](https://github.com/nyn-05/lead_scraper/blob/main/data/leads.db): `data/leads.db`
- [CSV JSON Excel](https://github.com/nyn-05/lead_scraper/blob/main/lead_scraper/storage.py): `lead_scraper/storage.py`

### Other components

- **User**
- **Public sources**
- **Search providers**
- **Google Maps**

## Connections

- User → CLI dispatch: runs command (evidence: `README.md`)
- User → Streamlit dashboard: uses dashboard (evidence: `README.md`)
- CLI dispatch → Maps listings: dispatches Maps (evidence: `main.py`)
- CLI dispatch → Social profiles: dispatches social (evidence: `main.py`)
- CLI dispatch → Photography collectors: runs collectors (evidence: `main.py`)
- Streamlit dashboard → Maps listings: starts collection (evidence: `lead_scraper/dashboard.py`)
- Streamlit dashboard → Social profiles: starts collection (evidence: `lead_scraper/dashboard.py`)
- Streamlit dashboard → Photography collectors: starts collection (evidence: `lead_scraper/dashboard.py`)
- Maps listings → Google Maps: collects listings (evidence: `README.md`)
- Social profiles → Search providers: searches profiles (evidence: `lead_scraper/scrapers.py`)
- Photography collectors → Public sources: collects posts (evidence: `README.md`)
- CLI dispatch → Configuration: loads settings (evidence: `main.py`)
- CLI dispatch → Post normalization: normalizes posts (evidence: `main.py`)
- CLI dispatch → Lead classification: classifies posts (evidence: `main.py`)
- CLI dispatch → Lead scoring: scores leads (evidence: `main.py`)
- Photography collectors → Post normalization: submits posts (evidence: `README.md`)
- Post normalization → Lead classification: feeds normalized posts (evidence: `README.md`)
- Lead classification → Lead scoring: provides classification (evidence: `lead_scraper/processing.py`)
- CLI dispatch → Database and exports: persists results (evidence: `main.py`)
- Maps listings → Database and exports: stores listings (evidence: `README.md`)
- Social profiles → Database and exports: stores profiles (evidence: `README.md`)
- Streamlit dashboard → Database and exports: reads stored data (evidence: `lead_scraper/dashboard.py`)
- Database and exports → SQLite database: reads and writes (evidence: `lead_scraper/storage.py`)
- Database and exports → CSV JSON Excel: writes exports (evidence: `lead_scraper/storage.py`)
