# gisblog/nih-covid19 architecture

> Architecture diagram of the GitHub repository gisblog/nih-covid19, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gisblog/nih-covid19
- Repository: https://github.com/gisblog/nih-covid19

## Overview

- Data Sources & External Services:
- Kaggle API: Highlight that the project uses the Kaggle API (installed via pip and configured with an API token stored in kaggle.json). This is the entry point which retrieves the competition data.
- Data Repository:
- Organized File Directories: Emphasize the file tree structure under the “covid19” directory that contains subdirectories (e.g., “kaggle/working/CORD-19-research-challenge/2020-03-13”) and further subgroupings (like “biorxiv_medrxiv”, “comm_use_subset”, “noncomm_use_subset”, “pmc_custom_license”). These represent different subsets of the COVID-19 research papers.
- Processing Modules:
- Python Scripts: Include components such as “kaggle.py” and “kaggle-geo.py”. These scripts are responsible for interfacing with the Kaggle API, processing downloaded JSON research papers, and potentially performing spatial analysis or other data transformations.
- Output/Submission Components:
- Answer Files: Represent the various answer JSONs (e.g., answers.task.0.biorxiv_medrxiv.json, answers.task.3.*.json) and the notebook submissions (as indicated in the README submission command) that are generated from the processing steps.

##### Relationships and Data Flow
- Flow of raw data into organized directories (grouped by research paper subsets) and show how the Python scripts pull data from these directories.
- That processed outputs (JSON answer files and other results) are generated and can be submitted back to Kaggle via the API.
- A feedback loop from the submission process if extra data analysis or iterative refinement is expected.

##### Architectural Patterns & Design Principles
- Modular data ingestion: Separate modules handle API communication and file organization.
- Clear separation of concerns: Distinct directories and scripts for raw data ingestion, processing, and output generation.
- Batch processing workflow: The system is designed for processing large volumes of files (JSON research papers) in an organized file structure.
- Use of configuration (kaggle.json) for secure API authentication.

##### Relevant Technologies & Frameworks
- Python: For scripting (kaggle.py, kaggle-geo.py) and processing logic.
- Kaggle API: The primary external interaction point.
- File system & JSON: Organization and storage of raw and processed data.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "External Services"
        api["Kaggle API"]:::external
    end

    subgraph "Data Ingestion Modules"
        ingestion1["kaggle.py"]:::module
        ingestion2["kaggle-geo.py"]:::module
    end

    subgraph "Raw Data Repository"
        rawMain["CORD-19 Research Challenge (2020-03-13)"]:::storage
        raw1["biorxiv_medrxiv"]:::storage
        raw2["comm_use_subset"]:::storage
        raw3["noncomm_use_subset"]:::storage
        raw4["pmc_custom_license"]:::storage
    end

    subgraph "Processed Papers JSON"
        proc1["papers.biorxiv_medrxiv.json"]:::process
        proc2["papers.comm_use_subset.json"]:::process
        proc3["papers.noncomm_use_subset.json"]:::process
        proc4["papers.pmc_custom_license.json"]:::process
    end

    subgraph "Output/Submission Modules"
        out1["answers.task.0.biorxiv_medrxiv.json"]:::output
        out2["answers.task.0.comm_use_subset.json"]:::output
        out3["answers.task.0.noncomm_use_subset.json"]:::output
        out4["answers.task.0.pmc_custom_license.json"]:::output
        out5["answers.task.3.biorxiv_medrxiv.json"]:::output
        out6["answers.task.3.comm_use_subset.json"]:::output
        out7["answers.task.3.noncomm_use_subset.json"]:::output
        out8["answers.task.3.pmc_custom_license.json"]:::output
    end

    %% Data Flow Connections
    api -->|"retrieves_data"| ingestion1
    api -->|"retrieves_data"| ingestion2

    ingestion1 -->|"stores_raw_JSON"| rawMain
    ingestion2 -->|"stores_raw_JSON"| rawMain

    rawMain -->|"contains_data"| raw1
    rawMain -->|"contains_data"| raw2
    rawMain -->|"contains_data"| raw3
    rawMain -->|"contains_data"| raw4

    rawMain -->|"sends_data"| proc1
    rawMain -->|"sends_data"| proc2
    rawMain -->|"sends_data"| proc3
    rawMain -->|"sends_data"| proc4

    proc1 -->|"generates_output"| out1
    proc2 -->|"generates_output"| out2
    proc3 -->|"generates_output"| out3
    proc4 -->|"generates_output"| out4

    proc1 -->|"generates_output"| out5
    proc2 -->|"generates_output"| out6
    proc3 -->|"generates_output"| out7
    proc4 -->|"generates_output"| out8

    out1 -->|"submits"| api
    out2 -->|"submits"| api
    out3 -->|"submits"| api
    out4 -->|"submits"| api
    out5 -->|"submits"| api
    out6 -->|"submits"| api
    out7 -->|"submits"| api
    out8 -->|"submits"| api

    %% Click Events
    click ingestion1 "https://github.com/gisblog/nih-covid19/blob/master/covid19/kaggle/working/kaggle.py"
    click ingestion2 "https://github.com/gisblog/nih-covid19/blob/master/covid19/kaggle/working/kaggle-geo.py"
    click rawMain "https://github.com/gisblog/nih-covid19/tree/master/covid19/kaggle/working/CORD-19-research-challenge/2020-03-13"
    click raw1 "https://github.com/gisblog/nih-covid19/tree/master/covid19/kaggle/working/CORD-19-research-challenge/2020-03-13/biorxiv_medrxiv"
    click raw2 "https://github.com/gisblog/nih-covid19/tree/master/covid19/kaggle/working/CORD-19-research-challenge/2020-03-13/comm_use_subset"
    click raw3 "https://github.com/gisblog/nih-covid19/tree/master/covid19/kaggle/working/CORD-19-research-challenge/2020-03-13/noncomm_use_subset"
    click raw4 "https://github.com/gisblog/nih-covid19/tree/master/covid19/kaggle/working/CORD-19-research-challenge/2020-03-13/pmc_custom_license"
    click proc1 "https://github.com/gisblog/nih-covid19/blob/master/covid19/kaggle/working/papers.biorxiv_medrxiv.json"
    click proc2 "https://github.com/gisblog/nih-covid19/blob/master/covid19/kaggle/working/papers.comm_use_subset.json"
    click proc3 "https://github.com/gisblog/nih-covid19/blob/master/covid19/kaggle/working/papers.noncomm_use_subset.json"
    click proc4 "https://github.com/gisblog/nih-covid19/blob/master/covid19/kaggle/working/papers.pmc_custom_license.json"
    click out1 "https://github.com/gisblog/nih-covid19/blob/master/covid19/kaggle/working/answers.task.0.biorxiv_medrxiv.json"
    click out2 "https://github.com/gisblog/nih-covid19/blob/master/covid19/kaggle/working/answers.task.0.comm_use_subset.json"
    click out3 "https://github.com/gisblog/nih-covid19/blob/master/covid19/kaggle/working/answers.task.0.noncomm_use_subset.json"
    click out4 "https://github.com/gisblog/nih-covid19/blob/master/covid19/kaggle/working/answers.task.0.pmc_custom_license.json"
    click out5 "https://github.com/gisblog/nih-covid19/blob/master/covid19/kaggle/working/answers.task.3.biorxiv_medrxiv.json"
    click out6 "https://github.com/gisblog/nih-covid19/blob/master/covid19/kaggle/working/answers.task.3.comm_use_subset.json"
    click out7 "https://github.com/gisblog/nih-covid19/blob/master/covid19/kaggle/working/answers.task.3.noncomm_use_subset.json"
    click out8 "https://github.com/gisblog/nih-covid19/blob/master/covid19/kaggle/working/answers.task.3.pmc_custom_license.json"

    %% Styles
    classDef external fill:#ffcc00,stroke:#000,stroke-width:2px;
    classDef module fill:#ccffcc,stroke:#000,stroke-width:2px;
    classDef storage fill:#cce5ff,stroke:#000,stroke-width:2px;
    classDef process fill:#ffcccc,stroke:#000,stroke-width:2px;
    classDef output fill:#ddccff,stroke:#000,stroke-width:2px;
```
