How greenelab/pubtator works
Main Components
• Entry Point: The execute.py file acts as the main entry point that accepts a configuration file (e.g., pubtator_central_config.json or pubtator_config.json) and triggers the processing.
• Configuration Manager: Derived from the config_files directory, this component holds JSON and markdown files that guide the behavior of the application (parameters such as input sources, processing options, etc.).
• Data Input and Storage:
• Data directory → Contains sample/example input data and temporary data storage (including files like sample annotations, tags and logs).
• Download directory → Involves downloading or logging the download process (e.g., bioconcepts2pubtator_offsets.gz.log).
• Processing and Pipeline Scripts:
• Scripts folder → Contains various scripts (download_full_text.py, extract_tags.py, hetnet_id_extractor.py, map_ids.py, pubtator_to_xml.py, utilities.py) that likely form the data processing pipeline. Each script likely represents a distinct processing step, such as converting formats or extracting specific tags.
• Mapper folder → Contains ID mapping logic (e.g., map_to_drugbank.py and drugbank_mapper.tsv) to connect the PubTator IDs with external drug identifiers (DrugBank).
• Testing/Verification: The tests.sh file suggests there’s a script to run some tests to verify functionality (possibly integration tests).
Relationships and Data Flow
• The process begins at execute.py with a chosen configuration (the JSON file provided).
• The Configuration Manager reads the parameters which determine how the pipeline proceeds—selecting source data, choosing the correct processing mode (PubTator vs. PubTator Central), etc.
• Data Acquisition: Data is either read from local directories (data/example) or downloaded (download folder) to be processed.
• Processing Pipeline: The various scripts in the scripts/ folder operate sequentially on the data:
• For example, download_full_text.py may fetch full-text data; then pubtator_to_xml.py transforms text annotations to XML format; extract_tags.py and hetnet_id_extractor.py extract specific data (like tags or hetnet IDs).
• Mapping: The mapper component (map_to_drugbank.py) maps IDs using the provided drugbank_mapper.tsv, thereby linking PubTator data to external identifiers.
• Logging/Temp Data: Throughout the process, temporary data and logs (e.g., in data/temp) are maintained for tracking batch processing.
• Final Output: After processing, the transformed annotation files are produced, which can then be consumed by external tools and pipelines (e.g., the snorkeling project).
Architectural Patterns and Design Principles
• Modular Pipeline: Each processing step is encapsulated in its own script making it easier to debug, test, or run individually.
• Configuration-Driven Behavior: The use of multiple configuration files ensures the system can adapt to different input types without altering the core code.
• Separation of Concerns: Data ingestion, processing, mapping, and logging are clearly separated into different directories, facilitating a clean maintenance structure.
Specific Inclusions
• The primary entry point (execute.py) and its reliance on configuration files.
• The modular and sequential nature of the processing pipeline (downloads, conversion, extraction, mapping).
• The file-based inputs/outputs and how the processed data integrates with external systems.
• The logging and temporary data paths that ensure traceability.