revtpark/vahan-scraper

Purpose and Entry Points

Vahan Scraper extracts vehicle-registration report data from India’s Vahan Dashboard. It offers interchangeable CLI entry points:

• Browser automation producing original XLSX downloads: scripts/scraper.py

• Direct PrimeFaces AJAX/HTTP replay producing parsed CSV: scripts/api.py

Both accept equivalent report-selection and geographic-scope arguments and target the same output directory hierarchy.

Source System Boundary

The system is a client-side data acquisition tool; its external dependency is the live Vahan Dashboard and its server-rendered/PrimeFaces AJAX behavior. Dashboard option lists, report configuration, state/RTO selection, and report data are all obtained at runtime rather than maintained locally.

• scripts/api.py

• scripts/scraper.py

Shared CLI Planning Model

Each command converts axis choices plus temporal and geographic options into a scrape plan. A run may represent:

• an all-states aggregate or a selected state;

• all RTOs, named RTOs, or aggregate state-level data;

• one year or an inclusive year range;

• a list-options discovery mode rather than extraction.

The main control-flow boundary is plan expansion: state/RTO/year combinations become independent report retrieval jobs, enabling partial completion and resume behavior.

• scripts/api.py

• scripts/scraper.py

Browser Retrieval Path

scripts/scraper.py drives Chromium through Playwright, interacting with dashboard controls as a user would. It is the fidelity-oriented path: the dashboard generates and downloads XLSX files directly. Browser startup and per-download interaction make this slower, but it preserves the source export format.

• scripts/scraper.py

• requirements.txt

HTTP/AJAX Retrieval Path

scripts/api.py avoids a browser by replaying the dashboard’s PrimeFaces AJAX protocol over HTTP. It obtains live UI/report state, submits the relevant selection changes and report request, then parses the returned HTML table into CSV. This is the performance-oriented path and depends more directly on undocumented upstream request/response structure.

• scripts/api.py

• requirements.txt

Output, Idempotency, and Failure Boundary

Both paths persist data below the configured output root using a deterministic scope hierarchy: state, RTO, and yaxis__xaxis, with one file per year. Browser jobs write .xlsx; HTTP jobs write .csv. Existing target files are skipped, so completed work is the durable checkpoint for interrupted or repeat runs.

• scripts/scraper.py

• scripts/api.py

Runtime and Dependencies

The project is Python CLI software. The HTTP path requires normal Python HTTP/HTML/CSV-related dependencies; the browser path additionally requires Playwright and a locally installed Chromium runtime. There is no repository-local service, database, or web application layer.

• requirements.txt

• scripts/api.py

• scripts/scraper.py