How eltebioinformatics/gmt_files_for_mulea works
Project type and purpose
• This is an open-source data‐centric repository that standardizes gene and protein sets into the GMT format.
• It provides both raw output (GMT_files/) and the ETL scripts (scripts_to_create_GMT_files/) that generate them from various external biological databases.
Main components
a. External Data Sources (APIs, downloads)
• Gene Ontology, PFAM, Reactome, WikiPathways, FlyAtlas, ModEncode, miRTarBase, TF databases (TFLink, TRRUST, ATRM, RegulonDB, Yeastract), Pathway Commons, etc.
b. ETL Pipelines (scripts_to_create_GMT_files/)
• Category-specific subfolders (GO/, PFAM/, Reactome/, WikiPathways/, TF/, miRNA/, Bands/, Ensembl-consecutive, ModEncode/, KEGG/, Pathway_Commons/, etc.)
• Languages and tools: Python, R, shell scripts, downloader utilities
• ID mapping submodule (scripts_to_create_GMT_files/ID_mapping_scripts/)
c. Data Store (GMT_files/)
• Species-organized directories, each containing multiple .gmt files by ontology, ID type, and filtering
d. Documentation & Metadata
• README.Rmd / README.md for overview, usage, and citation
• .gitignore, version control
e. Consumers
• mulea R package: read_gmt() to load files locally or from GitHub
• muleaData Bioconductor ExperimentHubData package for in-R access
Relationships and interactions
• ETL Pipelines → Data Store (generation of .gmt files)
• ID Mapping module feeds into multiple pipelines as needed
• Data Store → Consumers (R packages)
• Documentation touches every component (annotate processes, data, usage)
Architectural patterns and principles
• Modular ETL: each ontology category has its own folder and scripts
• Single responsibility: download, format, rename, convert in discrete scripts
• Data versioning: output files are static, regenerated only when scripts run
• Separation of concerns: code (scripts) vs. data (GMT_files/) vs. docs
Technologies and frameworks
• Python scripts (.py), R scripts (.R), shell scripts (.sh)
• Data formats: GMT (tab-delimited), CSV for mapping
• Version control: Git/GitHub
• R/Bioconductor for consumption
Elements
• Components: label External Sources, ETL Pipelines (with subcomponents), Data Store, ID Mapper, Documentation, Consumers
• External Sources → ETL Pipelines
• ID Mapper → ETL Pipelines
• ETL Pipelines → GMT_files
• GMT_files → R Consumers
• ETL modules: rectangles, orange
• ID mapping: parallelogram, yellow
• Consumers: rectangle with R logo, blue
Detailed capture of essential elements
• Break ETL Pipelines into category lanes: GO, PFAM, Reactome, TF, miRNA, etc.
• Inside each lane, list key scripts (e.g., GO.py & format_allgo.sh).
• A central orchestrator (could be manual or a higher-level shell script) triggering all pipelines.
• README and citation as global documentation.
• .gitignore and version control at the repository boundary.