How finverifyrag/dataset works
RegGuard is a resumable RBI/SEBI regulatory-document scraper. A user starts a run; the coordinator invokes RBI’s sequential-ID crawl and/or SEBI’s browser-paginated listing flow. Each regulator-specific path finds document details and PDFs, then shared helpers apply request politeness and record downloaded content and provenance. The README also documents CSV indexes and SEBI HTML fallbacks. The graph reflects the shown calls and documented outputs; unsampled implementation details are not inferred.
Main entry points: rbi_scraper.py RBI ID crawl · sebi_scraper.py SEBI page walker · common.py PDF downloader · run.py Run coordinator
Components 16
Run control
- Run coordinator
run.py
RBI collection
- RBI ID crawl
scraper/rbi_scraper.py - RBI detail parser
scraper/rbi_scraper.py
SEBI collection
- SEBI page walker
scraper/sebi_scraper.py - SEBI detail parser
scraper/sebi_scraper.py - HTML fallback
scraper/sebi_scraper.py
Shared download services
- HTTP and politeness
scraper/common.py - PDF downloader
scraper/common.py - Provenance manifest
scraper/common.py - PDF corpus
scraper/common.py - CSV indexes
scraper/rbi_scraper.py
Other components
- Operator
- Headless browser
- RBI website
- SEBI website
- M1 knowledge base
Connections 20
- Operator → Run coordinator: starts run
run.py - Run coordinator → RBI ID crawl: runs RBI
run.py - Run coordinator → SEBI page walker: runs SEBI
run.py - RBI ID crawl → RBI website: requests IDs
scraper/rbi_scraper.py - RBI ID crawl → RBI detail parser: parses details
scraper/rbi_scraper.py - RBI detail parser → RBI website: extracts PDF links
scraper/rbi_scraper.py - SEBI page walker → Headless browser: paginates with
scraper/sebi_scraper.py - Headless browser → SEBI website: loads listings
- SEBI page walker → SEBI detail parser: fetches details
scraper/sebi_scraper.py - SEBI detail parser → SEBI website: requests pages
scraper/sebi_scraper.py - RBI ID crawl → HTTP and politeness: uses session and limits
scraper/rbi_scraper.py - SEBI page walker → HTTP and politeness: uses session and limits
scraper/sebi_scraper.py - RBI ID crawl → PDF downloader: downloads PDFs
scraper/rbi_scraper.py - SEBI detail parser → PDF downloader: downloads PDFs
scraper/sebi_scraper.py - PDF downloader → PDF corpus: writes files
scraper/common.py - PDF downloader → Provenance manifest: records provenance
scraper/common.py - RBI ID crawl → CSV indexes: writes index
scraper/rbi_scraper.py - SEBI page walker → HTML fallback: writes fallback text
scraper/sebi_scraper.py - PDF corpus → M1 knowledge base: ingested by
README.md - Provenance manifest → M1 knowledge base: provides provenance
README.md