How gisblog/nih-covid19 works
• Data Sources & External Services:
• Kaggle API: Highlight that the project uses the Kaggle API (installed via pip and configured with an API token stored in kaggle.json). This is the entry point which retrieves the competition data.
• Data Repository:
• Organized File Directories: Emphasize the file tree structure under the “covid19” directory that contains subdirectories (e.g., “kaggle/working/CORD-19-research-challenge/2020-03-13”) and further subgroupings (like “biorxiv_medrxiv”, “comm_use_subset”, “noncomm_use_subset”, “pmc_custom_license”). These represent different subsets of the COVID-19 research papers.
• Processing Modules:
• Python Scripts: Include components such as “kaggle.py” and “kaggle-geo.py”. These scripts are responsible for interfacing with the Kaggle API, processing downloaded JSON research papers, and potentially performing spatial analysis or other data transformations.
• Output/Submission Components:
• Answer Files: Represent the various answer JSONs (e.g., answers.task.0.biorxiv_medrxiv.json, answers.task.3.*.json) and the notebook submissions (as indicated in the README submission command) that are generated from the processing steps.
Relationships and Data Flow
• Flow of raw data into organized directories (grouped by research paper subsets) and show how the Python scripts pull data from these directories.
• That processed outputs (JSON answer files and other results) are generated and can be submitted back to Kaggle via the API.
• A feedback loop from the submission process if extra data analysis or iterative refinement is expected.
Architectural Patterns & Design Principles
• Modular data ingestion: Separate modules handle API communication and file organization.
• Clear separation of concerns: Distinct directories and scripts for raw data ingestion, processing, and output generation.
• Batch processing workflow: The system is designed for processing large volumes of files (JSON research papers) in an organized file structure.
• Use of configuration (kaggle.json) for secure API authentication.
Relevant Technologies & Frameworks
• Python: For scripting (kaggle.py, kaggle-geo.py) and processing logic.
• Kaggle API: The primary external interaction point.
• File system & JSON: Organization and storage of raw and processed data.