How abhinav07-codes/govt-scheme-finder-agent works
This MVP acquires scheme pages from explicitly supplied official URLs, extracts and structures scheme information, processes linked guideline PDFs, and saves provenance-rich JSON. The CLI coordinates the workflow; source adapters fetch and parse portal pages, while shared extraction and processing modules clean HTML and identify sections. README-described optional OCR and India.gov.in browser rendering are preserved. Relationships are limited to calls and data handling visible in excerpts or README; unsampled logging and OCR internals are not inferred.
Main entry points: scraper.py Scraper CLI · base_scraper.py HTTP fetcher · myscheme_scraper.py myScheme adapter · india_gov_scraper.py India.gov adapter
Components 20
Acquisition
- Scraper CLI
scraper.py - myScheme adapter
scrapers/myscheme_scraper.py - India.gov adapter
scrapers/india_gov_scraper.py - HTTP fetcher
scrapers/base_scraper.py
Content extraction
- HTML extraction
extraction/html_extractor.py - HTML cleaning
processing/cleaner.py - Section detection
processing/section_detector.py - Scheme record builder
scrapers/myscheme_scraper.py
Guideline documents
- PDF processing
scraper.py - PDF text extraction
extraction/pdf_extractor.py - Optional OCR
extraction/ocr_extractor.py
Persistence and provenance
- Downloaded documents
data - Raw HTML
data/raw - Structured JSON
data/extracted - JSON writer
storage/json_storage.py - Content hashing
utils/hashing.py - Runtime configuration
config.py - Run logging
utils/logger.py
Other components
- URL operator
- Official portals
Connections 26
- URL operator → Scraper CLI: supplies URL
README.md - Scraper CLI → myScheme adapter: selects adapter
scraper.py - Scraper CLI → India.gov adapter: selects adapter
scraper.py - myScheme adapter → HTTP fetcher: fetches page
scrapers/myscheme_scraper.py - India.gov adapter → HTTP fetcher: fetches page
scrapers/india_gov_scraper.py - HTTP fetcher → Official portals: requests page
scrapers/base_scraper.py - myScheme adapter → HTML extraction: extracts HTML
scrapers/myscheme_scraper.py - India.gov adapter → HTML extraction: extracts HTML
scrapers/india_gov_scraper.py - HTML extraction → HTML cleaning: cleans soup
extraction/html_extractor.py - HTML extraction → Section detection: detects sections
extraction/html_extractor.py - Section detection → HTML cleaning: normalizes text
processing/section_detector.py - India.gov adapter → Scheme record builder: builds record
scrapers/india_gov_scraper.py - Scraper CLI → Scheme record builder: builds record
scraper.py - Scraper CLI → PDF processing: processes guidelines
scraper.py - PDF processing → HTTP fetcher: fetches PDFs
scraper.py - PDF processing → PDF text extraction: extracts PDF text
scraper.py - PDF processing → Downloaded documents: writes files
scraper.py - PDF text extraction → Optional OCR: optional OCR
README.md - Scraper CLI → Raw HTML: writes HTML
scraper.py - Scraper CLI → JSON writer: saves record
scraper.py - JSON writer → Structured JSON: writes JSON
storage/json_storage.py - myScheme adapter → Content hashing: hashes content
scrapers/myscheme_scraper.py - Scraper CLI → Content hashing: hashes PDFs
scraper.py - Scraper CLI → Runtime configuration: uses paths
scraper.py - HTTP fetcher → Runtime configuration: uses defaults
scrapers/base_scraper.py - Scraper CLI → Run logging: logs output
scraper.py