How greninja-tech/resumescraper works
1. This repo is a Python-based resume discovery pipeline with two front doors: a CLI batch runner in main.py and a local dashboard under webapp/. The core system is not a web application; the dashboard is a read/monitor interface around the scraping pipeline and its outputs.
2. Configuration is centralized in config/settings.py and the config/ package. That layer holds environment-driven secrets and runtime paths, especially API keys and the Tesseract OCR executable path. This creates a hard boundary between code and external credentials/tools.
3. Lead discovery starts in src/search_apis.py and src/github_scraper.py. These modules are responsible for finding candidate resume URLs or profile pages from external sources such as search APIs and GitHub, using city/location inputs from the CLI workflow.
4. Retrieved documents are handled by src/downloader.py, which sits between discovery and processing. Its role is network I/O plus local artifact acquisition: turning discovered links into files that can be parsed, OCR’d, and stored.
5. Text and field extraction happens in src/parser.py. It likely handles multiple input formats called out in the README: PDF, DOC, DOCX, TXT, and image/scanned documents via OCR. Tesseract is the key external dependency for non-text resumes.
6. src/validator.py and src/deduplicator.py form the data-quality layer. Validation normalizes extracted candidate records and checks completeness/format, while deduplication removes repeat candidates, repeated URLs, or overlapping resume records before export.
7. src/ai_enrichment.py is an optional intelligence/enrichment layer. It uses external AI and enrichment services to improve extracted records, infer missing candidate attributes, and augment items such as skills, experience, or contact details.
8. Output generation is handled by src/excel_exporter.py. This is the main persistence boundary of the pipeline: normalized candidate data is written into Excel reports and related output artifacts, which are stored under output/ according to the README.
9. Logging is centralized in src/logger.py. This is a cross-cutting concern used across the scrape/parse/enrich pipeline to make batch execution observable and to help diagnose missing APIs, OCR issues, or download failures.
10. The dashboard layer in webapp/server.py plus webapp/templates/index.html provides a local UI for viewing results and likely triggering or monitoring runs. It is separate from the core scraping modules and appears to consume generated output rather than directly managing the extraction logic.
11. Operationally, the system depends on several external services: SerpAPI/Bing for discovery, GitHub API for profile lookup, Anthropic for AI enrichment, Hunter for email enrichment, and Tesseract for OCR. These services materially shape the architecture because they determine which pipeline stages require network access, secrets, and local native tooling.
12. The overall data flow is: discover targets from external sources, download resume artifacts, parse/OCR them into structured text, validate and deduplicate records, enrich the candidate data, then export results and surface them in the dashboard. main.py ties these stages together into one batch-oriented workflow, with command-line city inputs controlling the scope of a run.
Main entry points: main.py Batch runner · ai_enrichment.py AI enrichment · logger.py Logger · settings.py Settings
Components 18
Core pipeline
Batch resume discovery, processing, and export stages.
- Batch runner · CLI entrypoint that wires the end-to-end pipeline together.
main.py - Search APIs · Finds candidate resume and profile URLs from external search sources.
src/search_apis.py - GitHub scraper · Discovers candidate profile or resume sources on GitHub.
src/github_scraper.py - Downloader · Fetches discovered links and stores local document artifacts.
src/downloader.py - Parser · Extracts text and fields from PDFs, Office docs, text files, and OCR inputs.
src/parser.py - Validator · Normalizes and checks completeness and format of extracted records.
src/validator.py - Deduplicator · Removes repeated candidates, URLs, and overlapping resume records.
src/deduplicator.py - AI enrichment · Augments records using external AI and enrichment services.
src/ai_enrichment.py - Excel export · Writes normalized candidate data to Excel reports and output artifacts.
src/excel_exporter.py - Logger · Central logging used across the pipeline for execution visibility and diagnostics.
src/logger.py
Config
Runtime settings, secrets, and external tool paths.
- Settings · Environment-driven runtime settings, secrets, and paths.
config/settings.py
Dashboard
Local read/monitor interface over generated outputs.
- Dashboard server · Serves the local dashboard for viewing run outputs and status.
webapp/server.py - Dashboard view · HTML template for the dashboard interface.
webapp/templates/index.html
External systems
APIs, OCR, and enrichment services used by the pipeline.
- Search APIs · External discovery services used for finding candidate pages and resumes.
- GitHub API · GitHub lookup and scraping surface for profile discovery.
- AI service · External model service used for record enrichment.
- Email service · External enrichment service for contact discovery.
- Tesseract OCR · Local OCR engine used for scanned or image-based resumes.
Connections 34
- Batch runner → Settings: loads · Reads runtime configuration and secrets.
- Batch runner → Search APIs: starts · Launches discovery based on CLI inputs.
- Batch runner → GitHub scraper: starts · Launches GitHub-based discovery.
- Batch runner → Downloader: pipes · Hands discovered targets to artifact acquisition.
- Batch runner → Parser: pipes · Advances downloaded files into text extraction.
- Batch runner → Validator: pipes · Routes extracted records into quality checks.
- Batch runner → Deduplicator: pipes · Runs duplicate removal in the batch flow.
- Batch runner → AI enrichment: pipes · Optionally enriches normalized records.
- Batch runner → Excel export: finishes · Writes final outputs for the batch run.
- Search APIs → Search APIs: queries · Uses external search providers to discover resume targets.
- GitHub scraper → GitHub API: queries · Uses GitHub lookup and scraping APIs.
- Search APIs → Downloader: targets · Passes discovered links for retrieval.
- GitHub scraper → Downloader: targets · Passes discovered profiles or files for retrieval.
- Downloader → Parser: hands off · Transfers downloaded artifacts for extraction.
- Parser → Tesseract OCR: ocr · Uses OCR for scanned or image-based documents.
- Parser → Validator: normalizes · Produces structured candidate records for validation.
- Validator → Deduplicator: filters · Sends cleaned records to duplicate removal.
- Deduplicator → AI enrichment: selects · Feeds unique records into enrichment.
- AI enrichment → AI service: calls · Uses AI assistance for record augmentation.
- AI enrichment → Email service: calls · Uses contact enrichment for missing details.
- AI enrichment → Excel export: outputs · Passes enriched records to report generation.
- Excel export → Dashboard server: publishes · Provides generated outputs for the dashboard to display.
- Dashboard server → Dashboard view: renders · Serves the dashboard template.
- Settings → Search APIs: configures · Supplies API keys and runtime options.
- Settings → Downloader: configures · Provides paths and download-related settings.
- Settings → Parser: configures · Provides OCR and parsing paths.
- Settings → AI enrichment: configures · Supplies enrichment credentials and service settings.
- Settings → Dashboard server: configures · Provides dashboard runtime settings.
- Logger → Batch runner: logs · Captures batch orchestration events.
- Logger → Downloader: logs · Captures retrieval and network failures.
- Logger → Parser: logs · Captures parsing and OCR issues.
- Logger → Validator: logs · Captures validation problems.
- Logger → AI enrichment: logs · Captures enrichment and API issues.
- Logger → Dashboard server: logs · Captures dashboard runtime events.