# greenelab/pubtator architecture

> Architecture diagram of the GitHub repository greenelab/pubtator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/greenelab/pubtator
- Repository: https://github.com/greenelab/pubtator

## Overview

##### Main Components
- Entry Point: The execute.py file acts as the main entry point that accepts a configuration file (e.g., pubtator_central_config.json or pubtator_config.json) and triggers the processing.
- Configuration Manager: Derived from the config_files directory, this component holds JSON and markdown files that guide the behavior of the application (parameters such as input sources, processing options, etc.).
- Data Input and Storage:
- Data directory → Contains sample/example input data and temporary data storage (including files like sample annotations, tags and logs).
- Download directory → Involves downloading or logging the download process (e.g., bioconcepts2pubtator_offsets.gz.log).
- Processing and Pipeline Scripts:
- Scripts folder → Contains various scripts (download_full_text.py, extract_tags.py, hetnet_id_extractor.py, map_ids.py, pubtator_to_xml.py, utilities.py) that likely form the data processing pipeline. Each script likely represents a distinct processing step, such as converting formats or extracting specific tags.
- Mapper folder → Contains ID mapping logic (e.g., map_to_drugbank.py and drugbank_mapper.tsv) to connect the PubTator IDs with external drug identifiers (DrugBank).
- Testing/Verification: The tests.sh file suggests there’s a script to run some tests to verify functionality (possibly integration tests).

##### Relationships and Data Flow
- The process begins at execute.py with a chosen configuration (the JSON file provided).
- The Configuration Manager reads the parameters which determine how the pipeline proceeds—selecting source data, choosing the correct processing mode (PubTator vs. PubTator Central), etc.
- Data Acquisition: Data is either read from local directories (data/example) or downloaded (download folder) to be processed.
- Processing Pipeline: The various scripts in the scripts/ folder operate sequentially on the data:
- For example, download_full_text.py may fetch full-text data; then pubtator_to_xml.py transforms text annotations to XML format; extract_tags.py and hetnet_id_extractor.py extract specific data (like tags or hetnet IDs).
- Mapping: The mapper component (map_to_drugbank.py) maps IDs using the provided drugbank_mapper.tsv, thereby linking PubTator data to external identifiers.
- Logging/Temp Data: Throughout the process, temporary data and logs (e.g., in data/temp) are maintained for tracking batch processing.
- Final Output: After processing, the transformed annotation files are produced, which can then be consumed by external tools and pipelines (e.g., the snorkeling project).

##### Architectural Patterns and Design Principles
- Modular Pipeline: Each processing step is encapsulated in its own script making it easier to debug, test, or run individually.
- Configuration-Driven Behavior: The use of multiple configuration files ensures the system can adapt to different input types without altering the core code.
- Separation of Concerns: Data ingestion, processing, mapping, and logging are clearly separated into different directories, facilitating a clean maintenance structure.

##### Specific Inclusions

- The primary entry point (execute.py) and its reliance on configuration files.
- The modular and sequential nature of the processing pipeline (downloads, conversion, extraction, mapping).
- The file-based inputs/outputs and how the processed data integrates with external systems.
- The logging and temporary data paths that ensure traceability.

## Diagram (Mermaid)

```mermaid
flowchart TD
    A["Entry Point (execute.py)"]:::entry
    B["Configuration Manager (config_files)"]:::config
    subgraph "Data Storage"
        C["Data Input (data/ - example & temp)"]:::datastore
        D["Download Logs (download/)"]:::datastore
    end
    E["Processing Pipeline Scripts (scripts/)"]:::process
    F["ID Mapping (mapper/)"]:::process
    G["External Consumers (greenelab/snorkeling)"]:::external
    H["Testing & Verification (tests.sh)"]:::testing

    %% Data flow
    A -->|"reads_config"| B
    B -->|"loads_parameters"| C
    C -->|"provides_data"| E
    D -->|"download_logs"| E
    E -->|"processes_data"| F
    F -->|"maps_IDs"| G
    E -->|"logs_output"| C
    H -.->|"verifies"| A

    %% Click Events
    click A "https://github.com/greenelab/pubtator/blob/master/execute.py"
    click B "https://github.com/greenelab/pubtator/tree/master/config_files/"
    click C "https://github.com/greenelab/pubtator/tree/master/data/"
    click D "https://github.com/greenelab/pubtator/tree/master/download/"
    click E "https://github.com/greenelab/pubtator/tree/master/scripts/"
    click F "https://github.com/greenelab/pubtator/tree/master/mapper/"
    click H "https://github.com/greenelab/pubtator/blob/master/tests.sh"

    %% Styles
    classDef entry fill:#f9e79f,stroke:#e67e22,stroke-width:2px;
    classDef config fill:#aed6f1,stroke:#3498db,stroke-width:2px;
    classDef datastore fill:#d5f5e3,stroke:#27ae60,stroke-width:2px;
    classDef process fill:#fadbd8,stroke:#e74c3c,stroke-width:2px;
    classDef external fill:#e8daef,stroke:#8e44ad,stroke-width:2px;
    classDef testing fill:#f4d03f,stroke:#f1c40f,stroke-width:2px;
```
