# petermr/pygetpapers architecture

> Architecture diagram of the GitHub repository petermr/pygetpapers, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/petermr/pygetpapers
- Repository: https://github.com/petermr/pygetpapers

## Overview

##### Project Type and Purpose
- This is a command‐line tool built in Python that assists text miners by querying open-access scientific repositories (e.g., Europe PMC, Crossref, arXiv, BioRxiv, MedRxiv, Rxivist) to download research paper metadata and full texts.
- Its main functionality includes constructing queries, interacting with various external APIs, downloading files (PDFs, XMLs), and generating metadata outputs in different formats (JSON, CSV, HTML).

##### File Structure for Components and Patterns
- The top-level directories include:
– “pygetpapers”: Contains the core code (including __init__.py and the main module pygetpapers.py).
– “pygetpapers/repository”: Holds individual modules (arxiv.py, crossref.py, europe_pmc.py, openalex.py, rxiv.py, rxivist.py) for repository-specific API implementations.
– “tests”: Contains test scripts.
– “docs”: Provides developer documentation and resources.
- The project follows a modular design with a “Repository Interface” (pygetpapers/repositoryinterface.py) which likely abstracts common behaviors across different repository modules, hinting at a strategy or factory design pattern.

##### README for Additional Insights
- The README explains how the CLI takes inputs (flags such as –q, –k, –o, –xml, –pdf, –update, etc.), builds queries, makes API requests, downloads full texts, and generates metadata files.

- Main Components to Illustrate:
a. Command-Line Interface (CLI):
– Entry point (pygetpapers command) that parses input arguments.
– Responsible for triggering the download process.
b. Core Application Logic:
– Central module (pygetpapers/pygetpapers.py) that orchestrates processes.
– Handles query formation, logging, configuration processing (config.ini, saved queries), and error handling.
c. Repository Layer:
– Common Repository Interface (pygetpapers/repositoryinterface.py) which abstracts how each repository module (arxiv.py, crossref.py, etc.) works.
– Each repository module encapsulates the logic to call an external API, process the response, and format metadata.
d. External APIs:
– Represent the external systems (Europe PMC, Crossref, arXiv, etc.) with which HTTP requests are made.
e. File & Data Management:
– Modules for downloading files (XML, PDF, supplementary data) and saving metadata (JSON, CSV, HTML).
– The downloaded papers are organized into “CProjects” where each paper has its dedicated folder.
f. Logging & Configuration:
– Logging functionality (log levels, log files) that feed back into the user interface.
– Configuration persistence through INI files and stored queries.
g. Auxiliary Modules:
– Testing (tests), Documentation (docs), and additional resources (resources).

- Relationships/Interactions:
– The CLI sends parsed arguments to the Main Application Logic.
– The application logic delegates tasks to appropriate repository modules via the Repository Interface.
– Each repository module makes HTTP requests to its external API, retrieves paper metadata and file URLs, and returns the data.
– The Main Application Logic then routes the data to the File Management unit to download full texts, produce metadata files (CSV, HTML, JSON), and organize papers into CProject structures.
– Logging and configuration are used throughout the process to record process steps and persist query settings.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interface Layer
    subgraph "User Interface"
        CLI("CLI Interface"):::cli
        AC("Application Controller"):::logic
    end
    CLI -->|"parseArgs"| AC

    %% Repository Layer
    subgraph "Repository Layer"
        RI("Repository Interface"):::repo
        AC -->|"delegateQuery"| RI
        subgraph "Repository Modules"
            A("arXiv Module"):::repo
            C("Crossref Module"):::repo
            E("EuropePMC Module"):::repo
            O("OpenAlex Module"):::repo
            R("Rxiv Module"):::repo
            X("Rxivist Module"):::repo
        end
        RI -->|"calls"| A
        RI -->|"calls"| C
        RI -->|"calls"| E
        RI -->|"calls"| O
        RI -->|"calls"| R
        RI -->|"calls"| X
    end

    %% External APIs Layer
    subgraph "External APIs"
        API_A("arXiv API"):::external
        API_C("Crossref API"):::external
        API_E("EuropePMC API"):::external
        API_O("OpenAlex API"):::external
        API_R("Rxiv API"):::external
        API_X("Rxivist API"):::external
    end
    A -->|"API_calls"| API_A
    C -->|"API_calls"| API_C
    E -->|"API_calls"| API_E
    O -->|"API_calls"| API_O
    R -->|"API_calls"| API_R
    X -->|"API_calls"| API_X

    %% Data & Persistence Layer
    subgraph "Data & Persistence"
        FM("File Manager/Output Generator"):::persistence
        CP("CProjects Directory"):::persistence
        CFG("Configuration (config.ini)"):::persistence
        EXH("Exception Handling (pgexceptions.py)"):::persistence
    end
    AC -->|"download&generate"| FM
    FM -->|"savesOutput"| CP
    AC -->|"logsConfig"| CFG
    AC -->|"errorHandling"| EXH

    %% Auxiliary Modules
    subgraph "Auxiliary Modules"
        TS("Testing Suite"):::auxiliary
        DOC("Documentation"):::auxiliary
    end

    %% Click Events
    click CLI "https://github.com/petermr/pygetpapers/blob/main/pygetpapers/pygetpapers.py"
    click AC "https://github.com/petermr/pygetpapers/blob/main/pygetpapers/pygetpapers.py"
    click RI "https://github.com/petermr/pygetpapers/blob/main/pygetpapers/repositoryinterface.py"
    click A "https://github.com/petermr/pygetpapers/blob/main/pygetpapers/repository/arxiv.py"
    click C "https://github.com/petermr/pygetpapers/blob/main/pygetpapers/repository/crossref.py"
    click E "https://github.com/petermr/pygetpapers/blob/main/pygetpapers/repository/europe_pmc.py"
    click O "https://github.com/petermr/pygetpapers/blob/main/pygetpapers/repository/openalex.py"
    click R "https://github.com/petermr/pygetpapers/blob/main/pygetpapers/repository/rxiv.py"
    click X "https://github.com/petermr/pygetpapers/blob/main/pygetpapers/repository/rxivist.py"
    click FM "https://github.com/petermr/pygetpapers/blob/main/pygetpapers/download_tools.py"
    click CFG "https://github.com/petermr/pygetpapers/blob/main/pygetpapers/config.ini"
    click EXH "https://github.com/petermr/pygetpapers/blob/main/pygetpapers/pgexceptions.py"
    click TS "https://github.com/petermr/pygetpapers/tree/main/tests"
    click DOC "https://github.com/petermr/pygetpapers/tree/main/docs"

    %% Styles
    classDef cli fill:#f9c,stroke:#333,stroke-width:2px;
    classDef logic fill:#bbf,stroke:#333,stroke-width:2px;
    classDef repo fill:#cfc,stroke:#333,stroke-width:2px;
    classDef external fill:#fc9,stroke:#333,stroke-width:2px;
    classDef persistence fill:#cff,stroke:#333,stroke-width:2px;
    classDef auxiliary fill:#ddd,stroke:#333,stroke-width:2px;
```
