# finverifyrag/dataset architecture

> Architecture diagram of the GitHub repository finverifyrag/dataset, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/finverifyrag/dataset
- Repository: https://github.com/finverifyrag/dataset

## Overview

RegGuard is a resumable RBI/SEBI regulatory-document scraper. A user starts a run; the coordinator invokes RBI’s sequential-ID crawl and/or SEBI’s browser-paginated listing flow. Each regulator-specific path finds document details and PDFs, then shared helpers apply request politeness and record downloaded content and provenance. The README also documents CSV indexes and SEBI HTML fallbacks. The graph reflects the shown calls and documented outputs; unsampled implementation details are not inferred.

Main entry points: [`scraper/rbi_scraper.py`](https://github.com/finverifyrag/dataset/blob/main/scraper/rbi_scraper.py) (RBI ID crawl), [`scraper/sebi_scraper.py`](https://github.com/finverifyrag/dataset/blob/main/scraper/sebi_scraper.py) (SEBI page walker), [`scraper/common.py`](https://github.com/finverifyrag/dataset/blob/main/scraper/common.py) (PDF downloader), [`run.py`](https://github.com/finverifyrag/dataset/blob/main/run.py) (Run coordinator)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_control["Run control"]
  node_runner["Run coordinator<br/>[run.py]"]
end

subgraph group_rbi["RBI collection"]
  node_rbi_crawl["RBI ID crawl<br/>[rbi_scraper.py]"]
  node_rbi_docs["RBI detail parser<br/>[rbi_scraper.py]"]
end

subgraph group_sebi["SEBI collection"]
  node_sebi_pages["SEBI page walker<br/>[sebi_scraper.py]"]
  node_sebi_docs["SEBI detail parser<br/>[sebi_scraper.py]"]
  node_html_output["HTML fallback<br/>[sebi_scraper.py]"]
end

subgraph group_shared["Shared download services"]
  node_http_policy["HTTP and politeness<br/>[common.py]"]
  node_pdf_fetch["PDF downloader<br/>[common.py]"]
  node_manifest[("Provenance manifest<br/>[common.py]")]
  node_pdf_output["PDF corpus<br/>[common.py]"]
  node_csv_output["CSV indexes<br/>[rbi_scraper.py]"]
end

node_operator(("Operator"))
node_browser["Headless browser"]
node_rbi_site["RBI website"]
node_sebi_site["SEBI website"]
node_m1["M1 knowledge base"]

node_operator -->|"starts run"| node_runner
node_runner -->|"runs RBI"| node_rbi_crawl
node_runner -->|"runs SEBI"| node_sebi_pages
node_rbi_crawl -->|"requests IDs"| node_rbi_site
node_rbi_crawl -->|"parses details"| node_rbi_docs
node_rbi_docs -->|"extracts PDF links"| node_rbi_site
node_sebi_pages -->|"paginates with"| node_browser
node_browser -->|"loads listings"| node_sebi_site
node_sebi_pages -->|"fetches details"| node_sebi_docs
node_sebi_docs -->|"requests pages"| node_sebi_site
node_rbi_crawl -->|"uses session and limits"| node_http_policy
node_sebi_pages -->|"uses session and limits"| node_http_policy
node_rbi_crawl -->|"downloads PDFs"| node_pdf_fetch
node_sebi_docs -->|"downloads PDFs"| node_pdf_fetch
node_pdf_fetch -->|"writes files"| node_pdf_output
node_pdf_fetch -->|"records provenance"| node_manifest
node_rbi_crawl -->|"writes index"| node_csv_output
node_sebi_pages -->|"writes fallback text"| node_html_output
node_pdf_output -->|"ingested by"| node_m1
node_manifest -->|"provides provenance"| node_m1

click node_runner "https://github.com/finverifyrag/dataset/blob/main/run.py"
click node_rbi_crawl "https://github.com/finverifyrag/dataset/blob/main/scraper/rbi_scraper.py"
click node_rbi_docs "https://github.com/finverifyrag/dataset/blob/main/scraper/rbi_scraper.py"
click node_sebi_pages "https://github.com/finverifyrag/dataset/blob/main/scraper/sebi_scraper.py"
click node_sebi_docs "https://github.com/finverifyrag/dataset/blob/main/scraper/sebi_scraper.py"
click node_http_policy "https://github.com/finverifyrag/dataset/blob/main/scraper/common.py"
click node_pdf_fetch "https://github.com/finverifyrag/dataset/blob/main/scraper/common.py"
click node_manifest "https://github.com/finverifyrag/dataset/blob/main/scraper/common.py"
click node_pdf_output "https://github.com/finverifyrag/dataset/blob/main/scraper/common.py"
click node_csv_output "https://github.com/finverifyrag/dataset/blob/main/scraper/rbi_scraper.py"
click node_html_output "https://github.com/finverifyrag/dataset/blob/main/scraper/sebi_scraper.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runner,node_browser toneBlue
class node_rbi_crawl,node_rbi_docs toneAmber
class node_sebi_pages,node_sebi_docs,node_html_output toneMint
class node_http_policy,node_pdf_fetch,node_manifest,node_pdf_output,node_csv_output toneRose
class node_operator,node_rbi_site,node_sebi_site,node_m1 toneIndigo
```

## Components

### Run control

- [Run coordinator](https://github.com/finverifyrag/dataset/blob/main/run.py): `run.py`

### RBI collection

- [RBI ID crawl](https://github.com/finverifyrag/dataset/blob/main/scraper/rbi_scraper.py): `scraper/rbi_scraper.py`
- [RBI detail parser](https://github.com/finverifyrag/dataset/blob/main/scraper/rbi_scraper.py): `scraper/rbi_scraper.py`

### SEBI collection

- [SEBI page walker](https://github.com/finverifyrag/dataset/blob/main/scraper/sebi_scraper.py): `scraper/sebi_scraper.py`
- [SEBI detail parser](https://github.com/finverifyrag/dataset/blob/main/scraper/sebi_scraper.py): `scraper/sebi_scraper.py`
- [HTML fallback](https://github.com/finverifyrag/dataset/blob/main/scraper/sebi_scraper.py): `scraper/sebi_scraper.py`

### Shared download services

- [HTTP and politeness](https://github.com/finverifyrag/dataset/blob/main/scraper/common.py): `scraper/common.py`
- [PDF downloader](https://github.com/finverifyrag/dataset/blob/main/scraper/common.py): `scraper/common.py`
- [Provenance manifest](https://github.com/finverifyrag/dataset/blob/main/scraper/common.py): `scraper/common.py`
- [PDF corpus](https://github.com/finverifyrag/dataset/blob/main/scraper/common.py): `scraper/common.py`
- [CSV indexes](https://github.com/finverifyrag/dataset/blob/main/scraper/rbi_scraper.py): `scraper/rbi_scraper.py`

### Other components

- **Operator**
- **Headless browser**
- **RBI website**
- **SEBI website**
- **M1 knowledge base**

## Connections

- Operator → Run coordinator: starts run (evidence: `run.py`)
- Run coordinator → RBI ID crawl: runs RBI (evidence: `run.py`)
- Run coordinator → SEBI page walker: runs SEBI (evidence: `run.py`)
- RBI ID crawl → RBI website: requests IDs (evidence: `scraper/rbi_scraper.py`)
- RBI ID crawl → RBI detail parser: parses details (evidence: `scraper/rbi_scraper.py`)
- RBI detail parser → RBI website: extracts PDF links (evidence: `scraper/rbi_scraper.py`)
- SEBI page walker → Headless browser: paginates with (evidence: `scraper/sebi_scraper.py`)
- Headless browser → SEBI website: loads listings
- SEBI page walker → SEBI detail parser: fetches details (evidence: `scraper/sebi_scraper.py`)
- SEBI detail parser → SEBI website: requests pages (evidence: `scraper/sebi_scraper.py`)
- RBI ID crawl → HTTP and politeness: uses session and limits (evidence: `scraper/rbi_scraper.py`)
- SEBI page walker → HTTP and politeness: uses session and limits (evidence: `scraper/sebi_scraper.py`)
- RBI ID crawl → PDF downloader: downloads PDFs (evidence: `scraper/rbi_scraper.py`)
- SEBI detail parser → PDF downloader: downloads PDFs (evidence: `scraper/sebi_scraper.py`)
- PDF downloader → PDF corpus: writes files (evidence: `scraper/common.py`)
- PDF downloader → Provenance manifest: records provenance (evidence: `scraper/common.py`)
- RBI ID crawl → CSV indexes: writes index (evidence: `scraper/rbi_scraper.py`)
- SEBI page walker → HTML fallback: writes fallback text (evidence: `scraper/sebi_scraper.py`)
- PDF corpus → M1 knowledge base: ingested by (evidence: `README.md`)
- Provenance manifest → M1 knowledge base: provides provenance (evidence: `README.md`)
