How saezlab/pypath works
Project type and overall purpose
• Type: Open-source Python library + CLI + optional web service.
• Purpose: Download, preprocess, integrate, and expose molecular biology resources (OmniPath databases, ID translation, homology, etc.) to Python programs, scripts, web clients, and external tools (R client, Cytoscape app, BEL exporter).
Break down the main architectural layers/components
• Inputs Layer
– Package: pypath.inputs
– Contains ~150 resource-specific modules (e.g. biogrid.py, chembl, disgenet/_api, etc.)
– Responsibilities: Download raw data from external sources (HTTP, FTP, REST), parse, clean, normalize, output standardized records.
• Data & Cache Layer
– Packages: pypath/share/cache.py, pypath/share/session.py, ~/.pypath/pickles/
– Responsibilities: Persist downloaded raw data, intermediate pickles, sqlite/SQL dumps; avoid repeated downloads; manage singleton instances (get_db).
• Core Database Layer
– Package: pypath.core
– Key classes: network, enz_sub, complex, annot, intercell
– Responsibilities: Take cleaned records, build in-memory data structures (igraph graphs, pandas DataFrames); provide API to query integrated databases.
• Utility Layer
– Package: pypath/utils and pypath/formats, pypath/reader
– Responsibilities: Standalone utilities (ID mapping, homology translation, GO processing, format I/O: OBO, XML, SDF, SQL dump, SQLite), readers for generic field and network operations.
• OmniPath Application Layer
– Package: pypath.omnipath
• omnipath.db: database manager — definitions of pickles, singleton pattern, building/loading logic.
• omnipath/app.py, omnipath/param.py, omnipath/export.py, bel.py, cellphonedb.py, etc.
– Subfolder omnipath/server: Sphinx-documented server generator & runner (run.py), HTML templates
– Responsibilities: Higher-level workflows:
• Build complete OmniPath databases
• Expose web service API
• Export to BEL or specialized formats
• Provide CLI scripts (scripts/server-runner.py, scripts/static-tables.py, etc.)
• Resources & Documentation Layer
– Folders: docs/, pypath/resources/, README.rst, Sphinx config
– Responsibilities: Build reference docs, tutorials (Jupyter notebooks), API reference, manual.
• External Clients & Integrations
– R: OmnipathR
– Cytoscape app
– BEL
– Direct Python import
– REST clients to pypath.omnipath.server
• Test & CI/CD
– Folder: test/
– GitHub Actions: .github/workflows/sphinx-build.yml
– Responsibilities: Unit tests for inputs, settings; doc build; version bump via .bumpversion.cfg.
Component relationships and data flows
1. External Resources → Inputs Layer (download & parse)
2. Inputs Layer → Data & Cache Layer (cache raw/intermediate files)
3. Data & Cache Layer → Core Database Layer (load/preprocess into graphs/DataFrames)
4. Core Database Layer → OmniPath Application Layer (DB manager picks classes from core)
5. OmniPath Application Layer →
a. Python API / CLI Scripts (user scripts, scripts/*.py)
b. Web Service (REST endpoints) → External Clients
c. Exports (BEL, Cytoscape, static tables)
– pypath.utils is called by Inputs, Core, and OmniPath layers wherever mapping, homology, or format conversions are needed.
– pypath.reader assists Core in graph/field abstraction.
Architectural patterns & design principles
• Modular Monolith: Well-isolated subpackages (core, inputs, utils, omnipath) but single codebase.
• Plugin-style Inputs: Each data source is a self-contained module following a common interface (input_module.py), enabling extensibility.
• Singleton / Factory: get_db methods and omnipath.db manage single instances of large database objects cached on disk.
• Cache-as-Database: Raw downloads and built pickles form a local data store to speed up rebuilds.
• Separation of Concerns:
• I/O & parsing in inputs
• Integration & in-memory models in core
• High-level workflows & APIs in omnipath
• Utilities in utils