# abhinav07-codes/govt-scheme-finder-agent architecture

> Architecture diagram of the GitHub repository abhinav07-codes/govt-scheme-finder-agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/abhinav07-codes/govt-scheme-finder-agent
- Repository: https://github.com/abhinav07-codes/govt-scheme-finder-agent

## Overview

This MVP acquires scheme pages from explicitly supplied official URLs, extracts and structures scheme information, processes linked guideline PDFs, and saves provenance-rich JSON. The CLI coordinates the workflow; source adapters fetch and parse portal pages, while shared extraction and processing modules clean HTML and identify sections. README-described optional OCR and India.gov.in browser rendering are preserved. Relationships are limited to calls and data handling visible in excerpts or README; unsampled logging and OCR internals are not inferred.

Main entry points: [`scraper.py`](https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/scraper.py) (Scraper CLI), [`scrapers/base_scraper.py`](https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/scrapers/base_scraper.py) (HTTP fetcher), [`scrapers/myscheme_scraper.py`](https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/scrapers/myscheme_scraper.py) (myScheme adapter), [`scrapers/india_gov_scraper.py`](https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/scrapers/india_gov_scraper.py) (India.gov adapter)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_acquisition["Acquisition"]
  node_cli["Scraper CLI<br/>[scraper.py]"]
  node_myscheme["myScheme adapter"]
  node_indiagov["India.gov adapter"]
  node_fetcher["HTTP fetcher<br/>[base_scraper.py]"]
end

subgraph group_content["Content extraction"]
  node_html["HTML extraction<br/>[html_extractor.py]"]
  node_cleaner["HTML cleaning<br/>[cleaner.py]"]
  node_sections["Section detection"]
  node_record["Scheme record builder"]
end

subgraph group_documents["Guideline documents"]
  node_pdfworkflow["PDF processing<br/>[scraper.py]"]
  node_pdf["PDF text extraction<br/>[pdf_extractor.py]"]
  node_ocr["Optional OCR<br/>[ocr_extractor.py]"]
end

subgraph group_persistence["Persistence and provenance"]
  node_documents[("Downloaded documents")]
  node_raw[("Raw HTML")]
  node_json[("Structured JSON")]
  node_storage["JSON writer<br/>[json_storage.py]"]
  node_hash["Content hashing<br/>[hashing.py]"]
  node_config["Runtime configuration<br/>[config.py]"]
  node_logger["Run logging<br/>[logger.py]"]
end

node_user(("URL operator"))
node_official(("Official portals"))

node_user -->|"supplies URL"| node_cli
node_cli -->|"selects adapter"| node_myscheme
node_cli -->|"selects adapter"| node_indiagov
node_myscheme -->|"fetches page"| node_fetcher
node_indiagov -->|"fetches page"| node_fetcher
node_fetcher -->|"requests page"| node_official
node_myscheme -->|"extracts HTML"| node_html
node_indiagov -->|"extracts HTML"| node_html
node_html -->|"cleans soup"| node_cleaner
node_html -->|"detects sections"| node_sections
node_sections -->|"normalizes text"| node_cleaner
node_indiagov -->|"builds record"| node_record
node_cli -->|"builds record"| node_record
node_cli -->|"processes guidelines"| node_pdfworkflow
node_pdfworkflow -->|"fetches PDFs"| node_fetcher
node_pdfworkflow -->|"extracts PDF text"| node_pdf
node_pdfworkflow -->|"writes files"| node_documents
node_pdf -.->|"optional OCR"| node_ocr
node_cli -->|"writes HTML"| node_raw
node_cli -->|"saves record"| node_storage
node_storage -->|"writes JSON"| node_json
node_myscheme -->|"hashes content"| node_hash
node_cli -->|"hashes PDFs"| node_hash
node_cli -->|"uses paths"| node_config
node_fetcher -->|"uses defaults"| node_config
node_cli -->|"logs output"| node_logger

click node_cli "https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/scraper.py"
click node_myscheme "https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/scrapers/myscheme_scraper.py"
click node_indiagov "https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/scrapers/india_gov_scraper.py"
click node_fetcher "https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/scrapers/base_scraper.py"
click node_html "https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/extraction/html_extractor.py"
click node_cleaner "https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/processing/cleaner.py"
click node_sections "https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/processing/section_detector.py"
click node_record "https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/scrapers/myscheme_scraper.py"
click node_pdfworkflow "https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/scraper.py"
click node_pdf "https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/extraction/pdf_extractor.py"
click node_ocr "https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/extraction/ocr_extractor.py"
click node_documents "https://github.com/abhinav07-codes/govt-scheme-finder-agent/tree/main/data"
click node_raw "https://github.com/abhinav07-codes/govt-scheme-finder-agent/tree/main/data/raw"
click node_json "https://github.com/abhinav07-codes/govt-scheme-finder-agent/tree/main/data/extracted"
click node_storage "https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/storage/json_storage.py"
click node_hash "https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/utils/hashing.py"
click node_config "https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/config.py"
click node_logger "https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/utils/logger.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_myscheme,node_indiagov,node_fetcher toneBlue
class node_html,node_cleaner,node_sections,node_record toneAmber
class node_pdfworkflow,node_pdf,node_ocr toneMint
class node_documents,node_raw,node_json,node_storage,node_hash,node_config,node_logger toneRose
class node_user,node_official toneIndigo
```

## Components

### Acquisition

- [Scraper CLI](https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/scraper.py): `scraper.py`
- [myScheme adapter](https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/scrapers/myscheme_scraper.py): `scrapers/myscheme_scraper.py`
- [India.gov adapter](https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/scrapers/india_gov_scraper.py): `scrapers/india_gov_scraper.py`
- [HTTP fetcher](https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/scrapers/base_scraper.py): `scrapers/base_scraper.py`

### Content extraction

- [HTML extraction](https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/extraction/html_extractor.py): `extraction/html_extractor.py`
- [HTML cleaning](https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/processing/cleaner.py): `processing/cleaner.py`
- [Section detection](https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/processing/section_detector.py): `processing/section_detector.py`
- [Scheme record builder](https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/scrapers/myscheme_scraper.py): `scrapers/myscheme_scraper.py`

### Guideline documents

- [PDF processing](https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/scraper.py): `scraper.py`
- [PDF text extraction](https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/extraction/pdf_extractor.py): `extraction/pdf_extractor.py`
- [Optional OCR](https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/extraction/ocr_extractor.py): `extraction/ocr_extractor.py`

### Persistence and provenance

- [Downloaded documents](https://github.com/abhinav07-codes/govt-scheme-finder-agent/tree/main/data): `data`
- [Raw HTML](https://github.com/abhinav07-codes/govt-scheme-finder-agent/tree/main/data/raw): `data/raw`
- [Structured JSON](https://github.com/abhinav07-codes/govt-scheme-finder-agent/tree/main/data/extracted): `data/extracted`
- [JSON writer](https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/storage/json_storage.py): `storage/json_storage.py`
- [Content hashing](https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/utils/hashing.py): `utils/hashing.py`
- [Runtime configuration](https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/config.py): `config.py`
- [Run logging](https://github.com/abhinav07-codes/govt-scheme-finder-agent/blob/main/utils/logger.py): `utils/logger.py`

### Other components

- **URL operator**
- **Official portals**

## Connections

- URL operator → Scraper CLI: supplies URL (evidence: `README.md`)
- Scraper CLI → myScheme adapter: selects adapter (evidence: `scraper.py`)
- Scraper CLI → India.gov adapter: selects adapter (evidence: `scraper.py`)
- myScheme adapter → HTTP fetcher: fetches page (evidence: `scrapers/myscheme_scraper.py`)
- India.gov adapter → HTTP fetcher: fetches page (evidence: `scrapers/india_gov_scraper.py`)
- HTTP fetcher → Official portals: requests page (evidence: `scrapers/base_scraper.py`)
- myScheme adapter → HTML extraction: extracts HTML (evidence: `scrapers/myscheme_scraper.py`)
- India.gov adapter → HTML extraction: extracts HTML (evidence: `scrapers/india_gov_scraper.py`)
- HTML extraction → HTML cleaning: cleans soup (evidence: `extraction/html_extractor.py`)
- HTML extraction → Section detection: detects sections (evidence: `extraction/html_extractor.py`)
- Section detection → HTML cleaning: normalizes text (evidence: `processing/section_detector.py`)
- India.gov adapter → Scheme record builder: builds record (evidence: `scrapers/india_gov_scraper.py`)
- Scraper CLI → Scheme record builder: builds record (evidence: `scraper.py`)
- Scraper CLI → PDF processing: processes guidelines (evidence: `scraper.py`)
- PDF processing → HTTP fetcher: fetches PDFs (evidence: `scraper.py`)
- PDF processing → PDF text extraction: extracts PDF text (evidence: `scraper.py`)
- PDF processing → Downloaded documents: writes files (evidence: `scraper.py`)
- PDF text extraction → Optional OCR: optional OCR (evidence: `README.md`)
- Scraper CLI → Raw HTML: writes HTML (evidence: `scraper.py`)
- Scraper CLI → JSON writer: saves record (evidence: `scraper.py`)
- JSON writer → Structured JSON: writes JSON (evidence: `storage/json_storage.py`)
- myScheme adapter → Content hashing: hashes content (evidence: `scrapers/myscheme_scraper.py`)
- Scraper CLI → Content hashing: hashes PDFs (evidence: `scraper.py`)
- Scraper CLI → Runtime configuration: uses paths (evidence: `scraper.py`)
- HTTP fetcher → Runtime configuration: uses defaults (evidence: `scrapers/base_scraper.py`)
- Scraper CLI → Run logging: logs output (evidence: `scraper.py`)
