# covid19india/data architecture

> Architecture diagram of the GitHub repository covid19india/data, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/covid19india/data
- Repository: https://github.com/covid19india/data

## Overview

#### Purpose and Published Surface
This repository is a batch data-publishing pipeline for COVID-19 India statistics, not an application server. It transforms Google Sheet-derived/raw inputs into static CSV datasets and V4 JSON feeds served at `data.covid19india.org`. The active consumer surface is V4 (`v4/min/data.min.json`, `v4/min/timeseries.min.json`) plus aggregated CSV; legacy JSON endpoints are deprecated.

#### Pipeline Orchestration
`main.sh` is the top-level batch entry point, sequencing generators and validation. The implementation spans Node.js and Python, with dependencies declared in `package.json`, `package-lock.json`, and `requirements.txt`. There is no persistent database or API runtime represented in the tree; generated static files are the delivery boundary.

Paths: `main.sh`, `package.json`, `requirements.txt`

#### Raw-Data Acquisition and Normalization
The pipeline consumes latest Google Sheet data and historical raw sheets, then converts sheet-shaped data to JSON/CSV representations. Generic sheet conversion and raw-data assembly form the input boundary; raw datasets preserve granular records and sources but are explicitly not the preferred downstream interface.

Paths: `src/sheet-to-json_generic.js`, `src/sheets-to-csv.js`, `src/build_raw_data.py`

#### State and District Aggregation
Dedicated transforms derive state daily series and district-level outputs from normalized/raw inputs. These stages establish the main geographic aggregation boundary used by CSV consumers and downstream V4 assembly.

Paths: `src/states_daily_to_csv.js`, `src/district_data_generator.js`, `src/sanity_check.js`

#### V4 API Data Construction
The V4 parser produces the structured JSON model used by the frontend: current state/district statistics and historical state time series. This is the primary product path; minified V4 files are the recommended externally consumed JSON artifacts.

Paths: `src/parser_v4.py`, `lib/constants.js`, `documentation/v4_data.md`

#### Supplemental Derived Datasets
Separate generators handle locale metadata and activity/change logs. These are adjacent published artifacts rather than core case aggregation, and likely consume the same generated/normalized data domain.

Paths: `src/generate_locale.js`, `src/generate_activity_log.js`, `src/generate_activity_log.py`

#### Validation and Operational Notification
Sanity checking is an explicit pre-publication quality gate for generated data. Telegram notification support indicates batch-job operational reporting; its invocation is likely coordinated through the shell workflow or Node entry module.

Paths: `src/sanity_check.js`, `lib/notify_tg.js`, `lib/index.js`

#### Infrastructure and External Boundaries
External dependencies are Google Sheets as the near-real-time source (noted as 10–20 minutes delayed), public static hosting at `data.covid19india.org`, and Telegram for notifications. GitHub Actions provides CI for JavaScript via `.github/workflows/javascript.yml`; no deployment workflow is visible. Documentation defines the public CSV, raw-data, and V4 contracts.

Paths: `.github/workflows/javascript.yml`, `documentation/csv/index.md`, `documentation/rawdata.md`

Main entry points: [`src/build_raw_data.py`](https://github.com/covid19india/data/blob/main/src/build_raw_data.py) (Raw-data assembly), [`src/parser_v4.py`](https://github.com/covid19india/data/blob/main/src/parser_v4.py) (V4 parser), [`main.sh`](https://github.com/covid19india/data/blob/main/main.sh) (Batch orchestrator), [`documentation/v4_data.md`](https://github.com/covid19india/data/blob/main/documentation/v4_data.md) (Data contracts)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inputs["Sources & normalization"]
  node_sheets[("Google Sheets<br/>external source")]
  node_sheet_json["Generic sheet conversion<br/>Node transform"]
  node_sheet_csv["Sheet CSV conversion<br/>Node transform<br/>[sheets-to-csv.js]"]
  node_raw_build["Raw-data assembly<br/>Python transform<br/>[build_raw_data.py]"]
end

subgraph group_aggregation["Aggregation &amp; checks"]
  node_state_daily["State daily aggregation<br/>Node transform"]
  node_district_data["District aggregation<br/>Node transform"]
  node_sanity{{"Sanity check<br/>quality gate<br/>[sanity_check.js]"}}
end

subgraph group_v4["V4 product build"]
  node_v4_parser["V4 parser<br/>Python transform<br/>[parser_v4.py]"]
  node_v4_constants["V4 constants<br/>shared definitions<br/>[constants.js]"]
  node_locale["Locale generator<br/>Node transform<br/>[generate_locale.js]"]
  node_activity["Activity log generators<br/>Node/Python transforms"]
end

subgraph group_publication["Published artifacts"]
  node_raw_data["Raw datasets<br/>static data"]
  node_csv_data["Aggregated CSV<br/>static data"]
  node_v4_min["V4 minified feeds<br/>static JSON"]
  node_v4_timeseries["V4 time series feed<br/>static JSON"]
  node_hosting(("data.covid19india.org<br/>static hosting"))
end

subgraph group_operations["Operations &amp; contracts"]
  node_orchestrator{{"Batch orchestrator<br/>shell entry point<br/>[main.sh]"}}
  node_telegram(("Telegram notifications<br/>external service"))
  node_notification["Telegram notifier<br/>Node integration<br/>[notify_tg.js]"]
  node_ci["JavaScript CI<br/>GitHub Actions workflow<br/>[javascript.yml]"]
  node_contracts["Data contracts<br/>documentation<br/>[v4_data.md]"]
end

node_sheets -->|"ingests"| node_sheet_json
node_sheets -->|"exports"| node_sheet_csv
node_sheet_json -->|"normalizes"| node_raw_build
node_sheet_csv -->|"supplies"| node_raw_build
node_raw_build -->|"publishes"| node_raw_data
node_raw_build -->|"feeds"| node_state_daily
node_raw_build -->|"feeds"| node_district_data
node_state_daily -->|"writes"| node_csv_data
node_district_data -->|"writes"| node_csv_data
node_state_daily -->|"supplies series"| node_v4_parser
node_district_data -->|"supplies districts"| node_v4_parser
node_v4_constants -->|"defines model"| node_v4_parser
node_v4_parser -->|"writes"| node_v4_min
node_v4_parser -->|"writes"| node_v4_timeseries
node_sanity -.->|"approves"| node_v4_min
node_sanity -.->|"approves"| node_csv_data
node_locale -->|"supplements"| node_v4_min
node_activity -->|"supplements"| node_v4_min
node_v4_min -->|"served by"| node_hosting
node_v4_timeseries -->|"served by"| node_hosting
node_csv_data -->|"served by"| node_hosting
node_raw_data -->|"served by"| node_hosting
node_orchestrator -->|"runs"| node_raw_build
node_orchestrator -->|"runs"| node_sanity
node_orchestrator -->|"runs"| node_v4_parser
node_orchestrator -->|"reports through"| node_notification
node_notification -->|"sends"| node_telegram
node_ci -.->|"checks JavaScript pipeline"| node_orchestrator
node_contracts -.->|"documents"| node_v4_min
node_contracts -.->|"documents"| node_csv_data
node_contracts -.->|"documents"| node_raw_data

click node_sheet_json "https://github.com/covid19india/data/blob/main/src/sheet-to-json_generic.js"
click node_sheet_csv "https://github.com/covid19india/data/blob/main/src/sheets-to-csv.js"
click node_raw_build "https://github.com/covid19india/data/blob/main/src/build_raw_data.py"
click node_state_daily "https://github.com/covid19india/data/blob/main/src/states_daily_to_csv.js"
click node_district_data "https://github.com/covid19india/data/blob/main/src/district_data_generator.js"
click node_sanity "https://github.com/covid19india/data/blob/main/src/sanity_check.js"
click node_v4_parser "https://github.com/covid19india/data/blob/main/src/parser_v4.py"
click node_v4_constants "https://github.com/covid19india/data/blob/main/lib/constants.js"
click node_locale "https://github.com/covid19india/data/blob/main/src/generate_locale.js"
click node_activity "https://github.com/covid19india/data/blob/main/src/generate_activity_log.js"
click node_orchestrator "https://github.com/covid19india/data/blob/main/main.sh"
click node_notification "https://github.com/covid19india/data/blob/main/lib/notify_tg.js"
click node_ci "https://github.com/covid19india/data/blob/main/.github/workflows/javascript.yml"
click node_contracts "https://github.com/covid19india/data/blob/main/documentation/v4_data.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sheets,node_sheet_json,node_sheet_csv,node_raw_build toneBlue
class node_state_daily,node_district_data,node_sanity toneAmber
class node_v4_parser,node_v4_constants,node_locale,node_activity toneMint
class node_raw_data,node_csv_data,node_v4_min,node_v4_timeseries,node_hosting toneRose
class node_orchestrator,node_telegram,node_notification,node_ci,node_contracts toneIndigo
```

## Components

### Sources & normalization

- **Google Sheets**: external source · Near-real-time and historical source data.
- [Generic sheet conversion](https://github.com/covid19india/data/blob/main/src/sheet-to-json_generic.js): Node transform · `src/sheet-to-json_generic.js`
- [Sheet CSV conversion](https://github.com/covid19india/data/blob/main/src/sheets-to-csv.js): Node transform · `src/sheets-to-csv.js`
- [Raw-data assembly](https://github.com/covid19india/data/blob/main/src/build_raw_data.py): Python transform · Preserves granular source records. · `src/build_raw_data.py`

### Aggregation & checks

- [State daily aggregation](https://github.com/covid19india/data/blob/main/src/states_daily_to_csv.js): Node transform · `src/states_daily_to_csv.js`
- [District aggregation](https://github.com/covid19india/data/blob/main/src/district_data_generator.js): Node transform · `src/district_data_generator.js`
- [Sanity check](https://github.com/covid19india/data/blob/main/src/sanity_check.js): quality gate · Pre-publication generated-data validation. · `src/sanity_check.js`

### V4 product build

- [V4 parser](https://github.com/covid19india/data/blob/main/src/parser_v4.py): Python transform · Builds current statistics and historical state series. · `src/parser_v4.py`
- [V4 constants](https://github.com/covid19india/data/blob/main/lib/constants.js): shared definitions · `lib/constants.js`
- [Locale generator](https://github.com/covid19india/data/blob/main/src/generate_locale.js): Node transform · `src/generate_locale.js`
- [Activity log generators](https://github.com/covid19india/data/blob/main/src/generate_activity_log.js): Node/Python transforms · `src/generate_activity_log.js`

### Published artifacts

- **Raw datasets**: static data · Published but not the preferred downstream interface.
- **Aggregated CSV**: static data · Geographic datasets for CSV consumers.
- **V4 minified feeds**: static JSON · Primary JSON consumer surface.
- **V4 time series feed**: static JSON · Historical state time series.
- **data.covid19india.org**: static hosting

### Operations & contracts

- [Batch orchestrator](https://github.com/covid19india/data/blob/main/main.sh): shell entry point · Sequences generators and validation. · `main.sh`
- **Telegram notifications**: external service · Batch-job operational reporting.
- [Telegram notifier](https://github.com/covid19india/data/blob/main/lib/notify_tg.js): Node integration · `lib/notify_tg.js`
- [JavaScript CI](https://github.com/covid19india/data/blob/main/.github/workflows/javascript.yml): GitHub Actions workflow · `.github/workflows/javascript.yml`
- [Data contracts](https://github.com/covid19india/data/blob/main/documentation/v4_data.md): documentation · Defines CSV, raw-data, and V4 public contracts. · `documentation/v4_data.md`

## Connections

- Google Sheets → Generic sheet conversion: ingests
- Google Sheets → Sheet CSV conversion: exports
- Generic sheet conversion → Raw-data assembly: normalizes
- Sheet CSV conversion → Raw-data assembly: supplies
- Raw-data assembly → Raw datasets: publishes
- Raw-data assembly → State daily aggregation: feeds
- Raw-data assembly → District aggregation: feeds
- State daily aggregation → Aggregated CSV: writes
- District aggregation → Aggregated CSV: writes
- State daily aggregation → V4 parser: supplies series
- District aggregation → V4 parser: supplies districts
- V4 constants → V4 parser: defines model
- V4 parser → V4 minified feeds: writes
- V4 parser → V4 time series feed: writes
- Sanity check → V4 minified feeds: approves
- Sanity check → Aggregated CSV: approves
- Locale generator → V4 minified feeds: supplements
- Activity log generators → V4 minified feeds: supplements
- V4 minified feeds → data.covid19india.org: served by
- V4 time series feed → data.covid19india.org: served by
- Aggregated CSV → data.covid19india.org: served by
- Raw datasets → data.covid19india.org: served by
- Batch orchestrator → Raw-data assembly: runs
- Batch orchestrator → Sanity check: runs
- Batch orchestrator → V4 parser: runs
- Batch orchestrator → Telegram notifier: reports through
- Telegram notifier → Telegram notifications: sends
- JavaScript CI → Batch orchestrator: checks JavaScript pipeline
- Data contracts → V4 minified feeds: documents
- Data contracts → Aggregated CSV: documents
- Data contracts → Raw datasets: documents