# owid/covid-19-data architecture

> Architecture diagram of the GitHub repository owid/covid-19-data, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/owid/covid-19-data
- Repository: https://github.com/owid/covid-19-data

## Overview

This is OWID’s archived COVID-19 data-production repository. Its principal workflow is to collect and process COVID-related datasets, then generate internal and public outputs; the README distinguishes these production scripts from the final datasets consumed by users. The tree shows separate pipelines for cases and deaths, testing, hospitalizations, variants, vaccinations, and excess mortality, alongside shared cleaning and export code. Several domain modules were not sampled, so their internal calls and precise orchestration are not asserted; edges below rely on README statements or the source index.

Main entry points: [`scripts/src/cowidev/vax/utils/base.py`](https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/vax/utils/base.py) (Vaccination processing), [`scripts/src/cowidev/megafile/generate.py`](https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/megafile/generate.py) (Dataset generator), [`scripts/src/cowidev/utils/clean`](https://github.com/owid/covid-19-data/tree/master/scripts/src/cowidev/utils/clean) (Data cleaning), [`scripts/src/cowidev/testing`](https://github.com/owid/covid-19-data/tree/master/scripts/src/cowidev/testing) (Testing data)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_production["Dataset production"]
  node_scripts["Production scripts"]
  node_megafile["Dataset generator<br/>[generate.py]"]
  node_steps["Dataset steps"]
  node_exports["Dataset exports"]
  node_publicdata["Public datasets"]
end

subgraph group_domains["Data domains"]
  node_cases["Cases and deaths<br/>[cases_deaths.py]"]
  node_testing["Testing data"]
  node_hospital["Hospital data<br/>[etl.py]"]
  node_variants["Variant data<br/>[etl.py]"]
  node_excess["Excess mortality<br/>[etl.py]"]
end

subgraph group_vaccination["Vaccination data"]
  node_vaccination["Vaccination collection<br/>[ecdc.py]"]
  node_vaxbase["Vaccination processing<br/>[base.py]"]
  node_vaxfiles["Vaccination files<br/>[files.py]"]
  node_twitter["Manual social sources<br/>[base.py]"]
end

subgraph group_shared["Shared services"]
  node_paths["Path configuration<br/>[paths.py]"]
  node_cleaning["Data cleaning"]
  node_web["Web downloads<br/>[download.py]"]
  node_s3[("S3 storage<br/>[s3.py]")]
  node_population["Population enrichment<br/>[shared.py]"]
end

node_maintainer(("Data maintainer"))

node_maintainer -->|"runs pipelines"| node_scripts
node_scripts -->|"generates dataset"| node_megafile
node_megafile -->|"loads steps"| node_steps
node_megafile -->|"uses exports"| node_exports
node_exports -->|"writes datasets"| node_publicdata
node_vaxbase -->|"loads data"| node_s3
node_vaxbase -->|"uses files"| node_vaxfiles
node_vaxbase -->|"reads paths"| node_paths
node_vaxbase -->|"cleans dates"| node_cleaning
node_vaccination -.->|"uses processing"| node_vaxbase
node_twitter -->|"cleans metrics"| node_cleaning
node_testing -->|"cleans values"| node_cleaning
node_testing -->|"downloads sources"| node_web
node_population -.->|"enriches measures"| node_publicdata

click node_megafile "https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/megafile/generate.py"
click node_steps "https://github.com/owid/covid-19-data/tree/master/scripts/src/cowidev/megafile/steps"
click node_exports "https://github.com/owid/covid-19-data/tree/master/scripts/src/cowidev/megafile/export"
click node_cases "https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/cases_deaths.py"
click node_testing "https://github.com/owid/covid-19-data/tree/master/scripts/src/cowidev/testing"
click node_hospital "https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/hosp/etl.py"
click node_variants "https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/variants/etl.py"
click node_excess "https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/xm/etl.py"
click node_vaccination "https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/vax/batch/ecdc.py"
click node_vaxbase "https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/vax/utils/base.py"
click node_vaxfiles "https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/vax/utils/files.py"
click node_twitter "https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/vax/manual/twitter/base.py"
click node_paths "https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/utils/paths.py"
click node_cleaning "https://github.com/owid/covid-19-data/tree/master/scripts/src/cowidev/utils/clean"
click node_web "https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/utils/web/download.py"
click node_s3 "https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/utils/s3.py"
click node_population "https://github.com/owid/covid-19-data/blob/master/scripts/scripts/shared.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_scripts,node_megafile,node_steps,node_exports,node_publicdata toneBlue
class node_cases,node_testing,node_hospital,node_variants,node_excess toneAmber
class node_vaccination,node_vaxbase,node_vaxfiles,node_twitter toneMint
class node_paths,node_cleaning,node_web,node_s3,node_population toneRose
class node_maintainer toneIndigo
```

## Components

### Dataset production

- **Production scripts**
- [Dataset generator](https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/megafile/generate.py): `scripts/src/cowidev/megafile/generate.py`
- [Dataset steps](https://github.com/owid/covid-19-data/tree/master/scripts/src/cowidev/megafile/steps): `scripts/src/cowidev/megafile/steps`
- [Dataset exports](https://github.com/owid/covid-19-data/tree/master/scripts/src/cowidev/megafile/export): `scripts/src/cowidev/megafile/export`
- **Public datasets**

### Data domains

- [Cases and deaths](https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/cases_deaths.py): `scripts/src/cowidev/cases_deaths.py`
- [Testing data](https://github.com/owid/covid-19-data/tree/master/scripts/src/cowidev/testing): `scripts/src/cowidev/testing`
- [Hospital data](https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/hosp/etl.py): `scripts/src/cowidev/hosp/etl.py`
- [Variant data](https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/variants/etl.py): `scripts/src/cowidev/variants/etl.py`
- [Excess mortality](https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/xm/etl.py): `scripts/src/cowidev/xm/etl.py`

### Vaccination data

- [Vaccination collection](https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/vax/batch/ecdc.py): `scripts/src/cowidev/vax/batch/ecdc.py`
- [Vaccination processing](https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/vax/utils/base.py): `scripts/src/cowidev/vax/utils/base.py`
- [Vaccination files](https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/vax/utils/files.py): `scripts/src/cowidev/vax/utils/files.py`
- [Manual social sources](https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/vax/manual/twitter/base.py): `scripts/src/cowidev/vax/manual/twitter/base.py`

### Shared services

- [Path configuration](https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/utils/paths.py): `scripts/src/cowidev/utils/paths.py`
- [Data cleaning](https://github.com/owid/covid-19-data/tree/master/scripts/src/cowidev/utils/clean): `scripts/src/cowidev/utils/clean`
- [Web downloads](https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/utils/web/download.py): `scripts/src/cowidev/utils/web/download.py`
- [S3 storage](https://github.com/owid/covid-19-data/blob/master/scripts/src/cowidev/utils/s3.py): `scripts/src/cowidev/utils/s3.py`
- [Population enrichment](https://github.com/owid/covid-19-data/blob/master/scripts/scripts/shared.py): `scripts/scripts/shared.py`

### Other components

- **Data maintainer**

## Connections

- Data maintainer → Production scripts: runs pipelines (evidence: `README.md`)
- Production scripts → Dataset generator: generates dataset
- Dataset generator → Dataset steps: loads steps (evidence: `scripts/src/cowidev/megafile/steps/__init__.py`)
- Dataset generator → Dataset exports: uses exports (evidence: `scripts/src/cowidev/megafile/export/__init__.py`)
- Dataset exports → Public datasets: writes datasets
- Vaccination processing → S3 storage: loads data (evidence: `scripts/src/cowidev/vax/utils/base.py`)
- Vaccination processing → Vaccination files: uses files (evidence: `scripts/src/cowidev/vax/utils/base.py`)
- Vaccination processing → Path configuration: reads paths (evidence: `scripts/src/cowidev/vax/utils/base.py`)
- Vaccination processing → Data cleaning: cleans dates (evidence: `scripts/src/cowidev/vax/utils/base.py`)
- Vaccination collection → Vaccination processing: uses processing (evidence: `scripts/src/cowidev/vax/batch/ecdc.py`)
- Manual social sources → Data cleaning: cleans metrics (evidence: `scripts/src/cowidev/vax/manual/twitter/paraguay.py`)
- Testing data → Data cleaning: cleans values (evidence: `scripts/src/cowidev/testing/incremental/nicaragua.py`)
- Testing data → Web downloads: downloads sources (evidence: `scripts/src/cowidev/testing/incremental/nicaragua.py`)
- Population enrichment → Public datasets: enriches measures
