# greninja-tech/resumescraper architecture

> Architecture diagram of the GitHub repository greninja-tech/resumescraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/greninja-tech/resumescraper
- Repository: https://github.com/greninja-tech/resumescraper

## Overview

1. This repo is a Python-based resume discovery pipeline with two front doors: a CLI batch runner in `main.py` and a local dashboard under `webapp/`. The core system is not a web application; the dashboard is a read/monitor interface around the scraping pipeline and its outputs.

2. Configuration is centralized in `config/settings.py` and the `config/` package. That layer holds environment-driven secrets and runtime paths, especially API keys and the Tesseract OCR executable path. This creates a hard boundary between code and external credentials/tools.

3. Lead discovery starts in `src/search_apis.py` and `src/github_scraper.py`. These modules are responsible for finding candidate resume URLs or profile pages from external sources such as search APIs and GitHub, using city/location inputs from the CLI workflow.

4. Retrieved documents are handled by `src/downloader.py`, which sits between discovery and processing. Its role is network I/O plus local artifact acquisition: turning discovered links into files that can be parsed, OCR’d, and stored.

5. Text and field extraction happens in `src/parser.py`. It likely handles multiple input formats called out in the README: PDF, DOC, DOCX, TXT, and image/scanned documents via OCR. Tesseract is the key external dependency for non-text resumes.

6. `src/validator.py` and `src/deduplicator.py` form the data-quality layer. Validation normalizes extracted candidate records and checks completeness/format, while deduplication removes repeat candidates, repeated URLs, or overlapping resume records before export.

7. `src/ai_enrichment.py` is an optional intelligence/enrichment layer. It uses external AI and enrichment services to improve extracted records, infer missing candidate attributes, and augment items such as skills, experience, or contact details.

8. Output generation is handled by `src/excel_exporter.py`. This is the main persistence boundary of the pipeline: normalized candidate data is written into Excel reports and related output artifacts, which are stored under `output/` according to the README.

9. Logging is centralized in `src/logger.py`. This is a cross-cutting concern used across the scrape/parse/enrich pipeline to make batch execution observable and to help diagnose missing APIs, OCR issues, or download failures.

10. The dashboard layer in `webapp/server.py` plus `webapp/templates/index.html` provides a local UI for viewing results and likely triggering or monitoring runs. It is separate from the core scraping modules and appears to consume generated output rather than directly managing the extraction logic.

11. Operationally, the system depends on several external services: SerpAPI/Bing for discovery, GitHub API for profile lookup, Anthropic for AI enrichment, Hunter for email enrichment, and Tesseract for OCR. These services materially shape the architecture because they determine which pipeline stages require network access, secrets, and local native tooling.

12. The overall data flow is: discover targets from external sources, download resume artifacts, parse/OCR them into structured text, validate and deduplicate records, enrich the candidate data, then export results and surface them in the dashboard. `main.py` ties these stages together into one batch-oriented workflow, with command-line city inputs controlling the scope of a run.

Main entry points: [`main.py`](https://github.com/greninja-tech/resumescraper/blob/main/main.py) (Batch runner), [`src/ai_enrichment.py`](https://github.com/greninja-tech/resumescraper/blob/main/src/ai_enrichment.py) (AI enrichment), [`src/logger.py`](https://github.com/greninja-tech/resumescraper/blob/main/src/logger.py) (Logger), [`config/settings.py`](https://github.com/greninja-tech/resumescraper/blob/main/config/settings.py) (Settings)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_grp_core["Core pipeline"]
  node_main["Batch runner<br/>orchestrator<br/>[main.py]"]
  node_search_apis["Search APIs<br/>discovery<br/>[search_apis.py]"]
  node_github_scraper["GitHub scraper<br/>discovery<br/>[github_scraper.py]"]
  node_downloader["Downloader<br/>fetch<br/>[downloader.py]"]
  node_parser["Parser<br/>extraction<br/>[parser.py]"]
  node_validator["Validator<br/>quality<br/>[validator.py]"]
  node_deduplicator["Deduplicator<br/>quality<br/>[deduplicator.py]"]
  node_ai_enrichment["AI enrichment<br/>[ai_enrichment.py]"]
  node_excel_exporter["Excel export<br/>output<br/>[excel_exporter.py]"]
  node_logger["Logger<br/>observability<br/>[logger.py]"]
end

subgraph group_grp_config["Config"]
  node_settings["Settings<br/>config<br/>[settings.py]"]
end

subgraph group_grp_ui["Dashboard"]
  node_server["Dashboard server<br/>ui<br/>[server.py]"]
  node_index_html["Dashboard view<br/>template<br/>[index.html]"]
end

subgraph group_grp_ext["External systems"]
  node_serpapi(("Search APIs<br/>external api"))
  node_github_api(("GitHub API<br/>external api"))
  node_anthropic(("AI service<br/>external api"))
  node_hunter(("Email service<br/>external api"))
  node_tesseract(("Tesseract OCR<br/>native tool"))
end

node_main -->|"loads"| node_settings
node_main -->|"starts"| node_search_apis
node_main -->|"starts"| node_github_scraper
node_main -->|"pipes"| node_downloader
node_main -->|"pipes"| node_parser
node_main -->|"pipes"| node_validator
node_main -->|"pipes"| node_deduplicator
node_main -->|"pipes"| node_ai_enrichment
node_main -->|"finishes"| node_excel_exporter
node_search_apis -->|"queries"| node_serpapi
node_github_scraper -->|"queries"| node_github_api
node_search_apis -->|"targets"| node_downloader
node_github_scraper -->|"targets"| node_downloader
node_downloader -->|"hands off"| node_parser
node_parser -->|"ocr"| node_tesseract
node_parser -->|"normalizes"| node_validator
node_validator -->|"filters"| node_deduplicator
node_deduplicator -->|"selects"| node_ai_enrichment
node_ai_enrichment -->|"calls"| node_anthropic
node_ai_enrichment -->|"calls"| node_hunter
node_ai_enrichment -->|"outputs"| node_excel_exporter
node_excel_exporter -->|"publishes"| node_server
node_server -->|"renders"| node_index_html
node_settings -->|"configures"| node_search_apis
node_settings -->|"configures"| node_downloader
node_settings -->|"configures"| node_parser
node_settings -->|"configures"| node_ai_enrichment
node_settings -->|"configures"| node_server
node_logger -.->|"logs"| node_main
node_logger -.->|"logs"| node_downloader
node_logger -.->|"logs"| node_parser
node_logger -.->|"logs"| node_validator
node_logger -.->|"logs"| node_ai_enrichment
node_logger -.->|"logs"| node_server

click node_main "https://github.com/greninja-tech/resumescraper/blob/main/main.py"
click node_settings "https://github.com/greninja-tech/resumescraper/blob/main/config/settings.py"
click node_search_apis "https://github.com/greninja-tech/resumescraper/blob/main/src/search_apis.py"
click node_github_scraper "https://github.com/greninja-tech/resumescraper/blob/main/src/github_scraper.py"
click node_downloader "https://github.com/greninja-tech/resumescraper/blob/main/src/downloader.py"
click node_parser "https://github.com/greninja-tech/resumescraper/blob/main/src/parser.py"
click node_validator "https://github.com/greninja-tech/resumescraper/blob/main/src/validator.py"
click node_deduplicator "https://github.com/greninja-tech/resumescraper/blob/main/src/deduplicator.py"
click node_ai_enrichment "https://github.com/greninja-tech/resumescraper/blob/main/src/ai_enrichment.py"
click node_excel_exporter "https://github.com/greninja-tech/resumescraper/blob/main/src/excel_exporter.py"
click node_logger "https://github.com/greninja-tech/resumescraper/blob/main/src/logger.py"
click node_server "https://github.com/greninja-tech/resumescraper/blob/main/webapp/server.py"
click node_index_html "https://github.com/greninja-tech/resumescraper/blob/main/webapp/templates/index.html"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_search_apis,node_github_scraper,node_downloader,node_parser,node_validator,node_deduplicator,node_ai_enrichment,node_excel_exporter,node_logger toneBlue
class node_settings toneAmber
class node_server,node_index_html toneMint
class node_serpapi,node_github_api,node_anthropic,node_hunter,node_tesseract toneRose
```

## Components

### Core pipeline

Batch resume discovery, processing, and export stages.

- [Batch runner](https://github.com/greninja-tech/resumescraper/blob/main/main.py): orchestrator · CLI entrypoint that wires the end-to-end pipeline together. · `main.py`
- [Search APIs](https://github.com/greninja-tech/resumescraper/blob/main/src/search_apis.py): discovery · Finds candidate resume and profile URLs from external search sources. · `src/search_apis.py`
- [GitHub scraper](https://github.com/greninja-tech/resumescraper/blob/main/src/github_scraper.py): discovery · Discovers candidate profile or resume sources on GitHub. · `src/github_scraper.py`
- [Downloader](https://github.com/greninja-tech/resumescraper/blob/main/src/downloader.py): fetch · Fetches discovered links and stores local document artifacts. · `src/downloader.py`
- [Parser](https://github.com/greninja-tech/resumescraper/blob/main/src/parser.py): extraction · Extracts text and fields from PDFs, Office docs, text files, and OCR inputs. · `src/parser.py`
- [Validator](https://github.com/greninja-tech/resumescraper/blob/main/src/validator.py): quality · Normalizes and checks completeness and format of extracted records. · `src/validator.py`
- [Deduplicator](https://github.com/greninja-tech/resumescraper/blob/main/src/deduplicator.py): quality · Removes repeated candidates, URLs, and overlapping resume records. · `src/deduplicator.py`
- [AI enrichment](https://github.com/greninja-tech/resumescraper/blob/main/src/ai_enrichment.py): enrichment · Augments records using external AI and enrichment services. · `src/ai_enrichment.py`
- [Excel export](https://github.com/greninja-tech/resumescraper/blob/main/src/excel_exporter.py): output · Writes normalized candidate data to Excel reports and output artifacts. · `src/excel_exporter.py`
- [Logger](https://github.com/greninja-tech/resumescraper/blob/main/src/logger.py): observability · Central logging used across the pipeline for execution visibility and diagnostics. · `src/logger.py`

### Config

Runtime settings, secrets, and external tool paths.

- [Settings](https://github.com/greninja-tech/resumescraper/blob/main/config/settings.py): config · Environment-driven runtime settings, secrets, and paths. · `config/settings.py`

### Dashboard

Local read/monitor interface over generated outputs.

- [Dashboard server](https://github.com/greninja-tech/resumescraper/blob/main/webapp/server.py): ui · Serves the local dashboard for viewing run outputs and status. · `webapp/server.py`
- [Dashboard view](https://github.com/greninja-tech/resumescraper/blob/main/webapp/templates/index.html): template · HTML template for the dashboard interface. · `webapp/templates/index.html`

### External systems

APIs, OCR, and enrichment services used by the pipeline.

- **Search APIs**: external api · External discovery services used for finding candidate pages and resumes.
- **GitHub API**: external api · GitHub lookup and scraping surface for profile discovery.
- **AI service**: external api · External model service used for record enrichment.
- **Email service**: external api · External enrichment service for contact discovery.
- **Tesseract OCR**: native tool · Local OCR engine used for scanned or image-based resumes.

## Connections

- Batch runner → Settings: loads · Reads runtime configuration and secrets.
- Batch runner → Search APIs: starts · Launches discovery based on CLI inputs.
- Batch runner → GitHub scraper: starts · Launches GitHub-based discovery.
- Batch runner → Downloader: pipes · Hands discovered targets to artifact acquisition.
- Batch runner → Parser: pipes · Advances downloaded files into text extraction.
- Batch runner → Validator: pipes · Routes extracted records into quality checks.
- Batch runner → Deduplicator: pipes · Runs duplicate removal in the batch flow.
- Batch runner → AI enrichment: pipes · Optionally enriches normalized records.
- Batch runner → Excel export: finishes · Writes final outputs for the batch run.
- Search APIs → Search APIs: queries · Uses external search providers to discover resume targets.
- GitHub scraper → GitHub API: queries · Uses GitHub lookup and scraping APIs.
- Search APIs → Downloader: targets · Passes discovered links for retrieval.
- GitHub scraper → Downloader: targets · Passes discovered profiles or files for retrieval.
- Downloader → Parser: hands off · Transfers downloaded artifacts for extraction.
- Parser → Tesseract OCR: ocr · Uses OCR for scanned or image-based documents.
- Parser → Validator: normalizes · Produces structured candidate records for validation.
- Validator → Deduplicator: filters · Sends cleaned records to duplicate removal.
- Deduplicator → AI enrichment: selects · Feeds unique records into enrichment.
- AI enrichment → AI service: calls · Uses AI assistance for record augmentation.
- AI enrichment → Email service: calls · Uses contact enrichment for missing details.
- AI enrichment → Excel export: outputs · Passes enriched records to report generation.
- Excel export → Dashboard server: publishes · Provides generated outputs for the dashboard to display.
- Dashboard server → Dashboard view: renders · Serves the dashboard template.
- Settings → Search APIs: configures · Supplies API keys and runtime options.
- Settings → Downloader: configures · Provides paths and download-related settings.
- Settings → Parser: configures · Provides OCR and parsing paths.
- Settings → AI enrichment: configures · Supplies enrichment credentials and service settings.
- Settings → Dashboard server: configures · Provides dashboard runtime settings.
- Logger → Batch runner: logs · Captures batch orchestration events.
- Logger → Downloader: logs · Captures retrieval and network failures.
- Logger → Parser: logs · Captures parsing and OCR issues.
- Logger → Validator: logs · Captures validation problems.
- Logger → AI enrichment: logs · Captures enrichment and API issues.
- Logger → Dashboard server: logs · Captures dashboard runtime events.