How petermr/pygetpapers works
Project Type and Purpose
• This is a command‐line tool built in Python that assists text miners by querying open-access scientific repositories (e.g., Europe PMC, Crossref, arXiv, BioRxiv, MedRxiv, Rxivist) to download research paper metadata and full texts.
• Its main functionality includes constructing queries, interacting with various external APIs, downloading files (PDFs, XMLs), and generating metadata outputs in different formats (JSON, CSV, HTML).
File Structure for Components and Patterns
• The top-level directories include:
– “pygetpapers”: Contains the core code (including __init__.py and the main module pygetpapers.py).
– “pygetpapers/repository”: Holds individual modules (arxiv.py, crossref.py, europe_pmc.py, openalex.py, rxiv.py, rxivist.py) for repository-specific API implementations.
– “tests”: Contains test scripts.
– “docs”: Provides developer documentation and resources.
• The project follows a modular design with a “Repository Interface” (pygetpapers/repositoryinterface.py) which likely abstracts common behaviors across different repository modules, hinting at a strategy or factory design pattern.
README for Additional Insights
• The README explains how the CLI takes inputs (flags such as –q, –k, –o, –xml, –pdf, –update, etc.), builds queries, makes API requests, downloads full texts, and generates metadata files.
• Main Components to Illustrate:
a. Command-Line Interface (CLI):
– Entry point (pygetpapers command) that parses input arguments.
– Responsible for triggering the download process.
b. Core Application Logic:
– Central module (pygetpapers/pygetpapers.py) that orchestrates processes.
– Handles query formation, logging, configuration processing (config.ini, saved queries), and error handling.
c. Repository Layer:
– Common Repository Interface (pygetpapers/repositoryinterface.py) which abstracts how each repository module (arxiv.py, crossref.py, etc.) works.
– Each repository module encapsulates the logic to call an external API, process the response, and format metadata.
d. External APIs:
– Represent the external systems (Europe PMC, Crossref, arXiv, etc.) with which HTTP requests are made.
e. File & Data Management:
– Modules for downloading files (XML, PDF, supplementary data) and saving metadata (JSON, CSV, HTML).
– The downloaded papers are organized into “CProjects” where each paper has its dedicated folder.
f. Logging & Configuration:
– Logging functionality (log levels, log files) that feed back into the user interface.
– Configuration persistence through INI files and stored queries.
g. Auxiliary Modules:
– Testing (tests), Documentation (docs), and additional resources (resources).
• Relationships/Interactions:
– The CLI sends parsed arguments to the Main Application Logic.
– The application logic delegates tasks to appropriate repository modules via the Repository Interface.
– Each repository module makes HTTP requests to its external API, retrieves paper metadata and file URLs, and returns the data.
– The Main Application Logic then routes the data to the File Management unit to download full texts, produce metadata files (CSV, HTML, JSON), and organize papers into CProject structures.
– Logging and configuration are used throughout the process to record process steps and persist query settings.