# revtpark/vahan-scraper architecture

> Architecture diagram of the GitHub repository revtpark/vahan-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/revtpark/vahan-scraper
- Repository: https://github.com/revtpark/vahan-scraper

## Overview

#### Purpose and Entry Points
Vahan Scraper extracts vehicle-registration report data from India’s Vahan Dashboard. It offers interchangeable CLI entry points:
- Browser automation producing original XLSX downloads: `scripts/scraper.py`
- Direct PrimeFaces AJAX/HTTP replay producing parsed CSV: `scripts/api.py`

Both accept equivalent report-selection and geographic-scope arguments and target the same output directory hierarchy.

#### Source System Boundary
The system is a client-side data acquisition tool; its external dependency is the live Vahan Dashboard and its server-rendered/PrimeFaces AJAX behavior. Dashboard option lists, report configuration, state/RTO selection, and report data are all obtained at runtime rather than maintained locally.
- `scripts/api.py`
- `scripts/scraper.py`

#### Shared CLI Planning Model
Each command converts axis choices plus temporal and geographic options into a scrape plan. A run may represent:
- an all-states aggregate or a selected state;
- all RTOs, named RTOs, or aggregate state-level data;
- one year or an inclusive year range;
- a list-options discovery mode rather than extraction.

The main control-flow boundary is plan expansion: state/RTO/year combinations become independent report retrieval jobs, enabling partial completion and resume behavior.
- `scripts/api.py`
- `scripts/scraper.py`

#### Browser Retrieval Path
`scripts/scraper.py` drives Chromium through Playwright, interacting with dashboard controls as a user would. It is the fidelity-oriented path: the dashboard generates and downloads XLSX files directly. Browser startup and per-download interaction make this slower, but it preserves the source export format.
- `scripts/scraper.py`
- `requirements.txt`

#### HTTP/AJAX Retrieval Path
`scripts/api.py` avoids a browser by replaying the dashboard’s PrimeFaces AJAX protocol over HTTP. It obtains live UI/report state, submits the relevant selection changes and report request, then parses the returned HTML table into CSV. This is the performance-oriented path and depends more directly on undocumented upstream request/response structure.
- `scripts/api.py`
- `requirements.txt`

#### Output, Idempotency, and Failure Boundary
Both paths persist data below the configured output root using a deterministic scope hierarchy: state, RTO, and `yaxis__xaxis`, with one file per year. Browser jobs write `.xlsx`; HTTP jobs write `.csv`. Existing target files are skipped, so completed work is the durable checkpoint for interrupted or repeat runs.
- `scripts/scraper.py`
- `scripts/api.py`

#### Runtime and Dependencies
The project is Python CLI software. The HTTP path requires normal Python HTTP/HTML/CSV-related dependencies; the browser path additionally requires Playwright and a locally installed Chromium runtime. There is no repository-local service, database, or web application layer.
- `requirements.txt`
- `scripts/api.py`
- `scripts/scraper.py`

Main entry points: [`scripts/scraper.py`](https://github.com/revtpark/vahan-scraper/blob/main/scripts/scraper.py) (Browser CLI), [`scripts/api.py`](https://github.com/revtpark/vahan-scraper/blob/main/scripts/api.py) (HTTP CLI), [`requirements.txt`](https://github.com/revtpark/vahan-scraper/blob/main/requirements.txt) (Python dependencies)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_cli["CLI planning"]
  node_cli_options["Report and scope options<br/>CLI arguments"]
  node_scrape_plan["Scrape plan<br/>job plan"]
  node_live_options["Live dashboard state<br/>runtime metadata"]
end

subgraph group_browser["Browser retrieval"]
  node_browser_cli["Browser CLI<br/>Python CLI<br/>[scraper.py]"]
  node_playwright["Playwright Chromium<br/>browser runtime<br/>[requirements.txt]"]
  node_xlsx_export["Dashboard XLSX export<br/>file download<br/>[scraper.py]"]
end

subgraph group_http["HTTP/AJAX retrieval"]
  node_http_cli["HTTP CLI<br/>Python CLI<br/>[api.py]"]
  node_primefaces["PrimeFaces AJAX replay<br/>HTTP client flow<br/>[api.py]"]
  node_html_table["Returned HTML table<br/>report response<br/>[api.py]"]
  node_csv_export["Parsed CSV<br/>file output<br/>[api.py]"]
end

subgraph group_output["Output and resume"]
  node_output_root[("Output hierarchy<br/>file storage")]
  node_existing_files["Completed-file checkpoint<br/>idempotency guard"]
end

node_user(("Operator<br/>user"))
node_dashboard{{"Vahan Dashboard<br/>external system"}}
node_dependencies["Python dependencies<br/>runtime dependencies<br/>[requirements.txt]"]

node_user -->|"runs"| node_browser_cli
node_user -->|"runs"| node_http_cli
node_browser_cli -->|"parses"| node_cli_options
node_http_cli -->|"parses"| node_cli_options
node_cli_options -->|"expands into"| node_scrape_plan
node_browser_cli -->|"discovers"| node_live_options
node_http_cli -->|"discovers"| node_live_options
node_live_options -->|"loaded from"| node_dashboard
node_scrape_plan -->|"checks targets"| node_existing_files
node_browser_cli -->|"drives"| node_playwright
node_playwright -->|"interacts with"| node_dashboard
node_scrape_plan -->|"retrieves jobs"| node_xlsx_export
node_dashboard -->|"generates"| node_xlsx_export
node_http_cli -->|"uses"| node_primefaces
node_scrape_plan -->|"submits jobs"| node_primefaces
node_primefaces -->|"replays requests to"| node_dashboard
node_dashboard -->|"returns"| node_html_table
node_html_table -->|"parses to"| node_csv_export
node_xlsx_export -->|"writes"| node_output_root
node_csv_export -->|"writes"| node_output_root
node_output_root -->|"provides"| node_existing_files
node_dependencies -.->|"supports"| node_browser_cli
node_dependencies -.->|"supports"| node_http_cli

click node_browser_cli "https://github.com/revtpark/vahan-scraper/blob/main/scripts/scraper.py"
click node_http_cli "https://github.com/revtpark/vahan-scraper/blob/main/scripts/api.py"
click node_playwright "https://github.com/revtpark/vahan-scraper/blob/main/requirements.txt"
click node_xlsx_export "https://github.com/revtpark/vahan-scraper/blob/main/scripts/scraper.py"
click node_primefaces "https://github.com/revtpark/vahan-scraper/blob/main/scripts/api.py"
click node_html_table "https://github.com/revtpark/vahan-scraper/blob/main/scripts/api.py"
click node_csv_export "https://github.com/revtpark/vahan-scraper/blob/main/scripts/api.py"
click node_dependencies "https://github.com/revtpark/vahan-scraper/blob/main/requirements.txt"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli_options,node_scrape_plan,node_live_options toneBlue
class node_browser_cli,node_playwright,node_xlsx_export toneAmber
class node_http_cli,node_primefaces,node_html_table,node_csv_export toneMint
class node_output_root,node_existing_files toneRose
class node_user,node_dashboard,node_dependencies toneNeutral
```

## Components

### CLI planning

- **Report and scope options**: CLI arguments · Axes, year range, state, RTO, and list-options mode.
- **Scrape plan**: job plan · Expands state/RTO/year scope into independent retrieval jobs.
- **Live dashboard state**: runtime metadata · Report configuration and geographic options are discovered from Vahan.

### Browser retrieval

- [Browser CLI](https://github.com/revtpark/vahan-scraper/blob/main/scripts/scraper.py): Python CLI · Fidelity-oriented XLSX export entry point. · `scripts/scraper.py`
- [Playwright Chromium](https://github.com/revtpark/vahan-scraper/blob/main/requirements.txt): browser runtime · Automates dashboard controls and downloads. · `requirements.txt`
- [Dashboard XLSX export](https://github.com/revtpark/vahan-scraper/blob/main/scripts/scraper.py): file download · Original spreadsheet generated by the dashboard. · `scripts/scraper.py`

### HTTP/AJAX retrieval

- [HTTP CLI](https://github.com/revtpark/vahan-scraper/blob/main/scripts/api.py): Python CLI · Performance-oriented CSV extraction entry point. · `scripts/api.py`
- [PrimeFaces AJAX replay](https://github.com/revtpark/vahan-scraper/blob/main/scripts/api.py): HTTP client flow · Replays selection changes and report requests. · `scripts/api.py`
- [Returned HTML table](https://github.com/revtpark/vahan-scraper/blob/main/scripts/api.py): report response · `scripts/api.py`
- [Parsed CSV](https://github.com/revtpark/vahan-scraper/blob/main/scripts/api.py): file output · HTML report table serialized as CSV. · `scripts/api.py`

### Output and resume

- **Output hierarchy**: file storage · Deterministic state/RTO/yaxis__xaxis paths with one file per year.
- **Completed-file checkpoint**: idempotency guard · Existing target files are skipped for resume behavior.

### Other components

- **Operator**: user
- **Vahan Dashboard**: external system · Live server-rendered dashboard and PrimeFaces AJAX endpoint.
- [Python dependencies](https://github.com/revtpark/vahan-scraper/blob/main/requirements.txt): runtime dependencies · HTTP/HTML/CSV libraries plus Playwright for browser retrieval. · `requirements.txt`

## Connections

- Operator → Browser CLI: runs
- Operator → HTTP CLI: runs
- Browser CLI → Report and scope options: parses
- HTTP CLI → Report and scope options: parses
- Report and scope options → Scrape plan: expands into
- Browser CLI → Live dashboard state: discovers
- HTTP CLI → Live dashboard state: discovers
- Live dashboard state → Vahan Dashboard: loaded from
- Scrape plan → Completed-file checkpoint: checks targets
- Browser CLI → Playwright Chromium: drives
- Playwright Chromium → Vahan Dashboard: interacts with
- Scrape plan → Dashboard XLSX export: retrieves jobs
- Vahan Dashboard → Dashboard XLSX export: generates
- HTTP CLI → PrimeFaces AJAX replay: uses
- Scrape plan → PrimeFaces AJAX replay: submits jobs
- PrimeFaces AJAX replay → Vahan Dashboard: replays requests to
- Vahan Dashboard → Returned HTML table: returns
- Returned HTML table → Parsed CSV: parses to
- Dashboard XLSX export → Output hierarchy: writes
- Parsed CSV → Output hierarchy: writes
- Output hierarchy → Completed-file checkpoint: provides
- Python dependencies → Browser CLI: supports
- Python dependencies → HTTP CLI: supports