# eltebioinformatics/gmt_files_for_mulea architecture

> Architecture diagram of the GitHub repository eltebioinformatics/gmt_files_for_mulea, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/eltebioinformatics/gmt_files_for_mulea
- Repository: https://github.com/eltebioinformatics/gmt_files_for_mulea

## Overview

##### Project type and purpose
- This is an open-source data‐centric repository that standardizes gene and protein sets into the GMT format.
- It provides both raw output (GMT_files/) and the ETL scripts (scripts_to_create_GMT_files/) that generate them from various external biological databases.

##### Main components
a. External Data Sources (APIs, downloads)
- Gene Ontology, PFAM, Reactome, WikiPathways, FlyAtlas, ModEncode, miRTarBase, TF databases (TFLink, TRRUST, ATRM, RegulonDB, Yeastract), Pathway Commons, etc.
b. ETL Pipelines (scripts_to_create_GMT_files/)
- Category-specific subfolders (GO/, PFAM/, Reactome/, WikiPathways/, TF/, miRNA/, Bands/, Ensembl-consecutive, ModEncode/, KEGG/, Pathway_Commons/, etc.)
- Languages and tools: Python, R, shell scripts, downloader utilities
- ID mapping submodule (scripts_to_create_GMT_files/ID_mapping_scripts/)
c. Data Store (GMT_files/)
- Species-organized directories, each containing multiple .gmt files by ontology, ID type, and filtering
d. Documentation & Metadata
- README.Rmd / README.md for overview, usage, and citation
- .gitignore, version control
e. Consumers
- mulea R package: read_gmt() to load files locally or from GitHub
- muleaData Bioconductor ExperimentHubData package for in-R access

##### Relationships and interactions
- ETL Pipelines → Data Store (generation of .gmt files)
- ID Mapping module feeds into multiple pipelines as needed
- Data Store → Consumers (R packages)
- Documentation touches every component (annotate processes, data, usage)

##### Architectural patterns and principles
- Modular ETL: each ontology category has its own folder and scripts
- Single responsibility: download, format, rename, convert in discrete scripts
- Data versioning: output files are static, regenerated only when scripts run
- Separation of concerns: code (scripts) vs. data (GMT_files/) vs. docs

##### Technologies and frameworks
- Python scripts (.py), R scripts (.R), shell scripts (.sh)
- Data formats: GMT (tab-delimited), CSV for mapping
- Version control: Git/GitHub
- R/Bioconductor for consumption

##### Elements
- Components: label External Sources, ETL Pipelines (with subcomponents), Data Store, ID Mapper, Documentation, Consumers
- External Sources → ETL Pipelines
- ID Mapper → ETL Pipelines
- ETL Pipelines → GMT_files
- GMT_files → R Consumers
- ETL modules: rectangles, orange
- ID mapping: parallelogram, yellow
- Consumers: rectangle with R logo, blue

##### Detailed capture of essential elements
- Break ETL Pipelines into category lanes: GO, PFAM, Reactome, TF, miRNA, etc.
- Inside each lane, list key scripts (e.g., GO.py & format_allgo.sh).
- A central orchestrator (could be manual or a higher-level shell script) triggering all pipelines.
- README and citation as global documentation.
- .gitignore and version control at the repository boundary.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Sources
    subgraph "External Data Sources"
        ES["Various Public Bio DBs"]:::external
    end

    %% ID Mapping Module
    ID["ID Mapping Module"]:::idmap

    %% ETL Pipelines
    subgraph "ETL Pipelines" 
        direction TB
        GO["GO Pipeline"]:::etl
        PFAM["PFAM Pipeline"]:::etl
        Reactome["Reactome Pipeline"]:::etl
        WikiPathways["WikiPathways Pipeline"]:::etl
        FlyAtlas["FlyAtlas Pipeline"]:::etl
        ModEncode["ModEncode Pipeline"]:::etl
        miRNA["miRNA Pipeline"]:::etl
        TF["TF Pipeline"]:::etl
        KEGG["KEGG Pipeline"]:::etl
        Pathway_Commons["Pathway Commons Pipeline"]:::etl
        SignaLink["SignaLink Pipeline"]:::etl
        Bands["Chromosomal Bands Pipeline"]:::etl
        Ensembl["Ensembl Consecutive Genes Pipeline"]:::etl
    end

    %% Data Store
    DT[("GMT_files Directory")]:::datastore

    %% Consumers
    subgraph "Consumers"
        mulea["mulea R Package"]:::consumer
        muleaData["muleaData Bioconductor Package"]:::consumer
    end

    %% Documentation & Metadata
    subgraph "Documentation & Metadata"
        README["README.md / README.Rmd"]:::docs
        GITIGNORE[".gitignore"]:::docs
    end

    %% Relationships
    ES -->|ingest data| GO
    ES --> PFAM
    ES --> Reactome
    ES --> WikiPathways
    ES --> FlyAtlas
    ES --> ModEncode
    ES --> miRNA
    ES --> TF
    ES --> KEGG
    ES --> Pathway_Commons
    ES --> SignaLink
    ES --> Bands
    ES --> Ensembl

    ID -->|maps IDs for| GO
    ID --> PFAM
    ID --> Reactome
    ID --> WikiPathways
    ID --> FlyAtlas
    ID --> ModEncode
    ID --> miRNA
    ID --> TF
    ID --> KEGG
    ID --> Pathway_Commons
    ID --> SignaLink
    ID --> Bands
    ID --> Ensembl

    %% ETL to Data Store
    GO -->|"generates GMT"| DT
    PFAM --> DT
    Reactome --> DT
    WikiPathways --> DT
    FlyAtlas --> DT
    ModEncode --> DT
    miRNA --> DT
    TF --> DT
    KEGG --> DT
    Pathway_Commons --> DT
    SignaLink --> DT
    Bands --> DT
    Ensembl --> DT

    %% Data Store to Consumers
    DT -->|"loads GMT"| mulea
    DT --> muleaData

    %% Docs connections
    README --- ES
    README --- ID
    README --- GO
    README --- DT
    GITIGNORE --- ES
    GITIGNORE --- GO

    %% Click Events
    click GO "https://github.com/eltebioinformatics/gmt_files_for_mulea/tree/main/scripts_to_create_GMT_files/GO"
    click PFAM "https://github.com/eltebioinformatics/gmt_files_for_mulea/tree/main/scripts_to_create_GMT_files/PFAM"
    click Reactome "https://github.com/eltebioinformatics/gmt_files_for_mulea/tree/main/scripts_to_create_GMT_files/Reactome"
    click WikiPathways "https://github.com/eltebioinformatics/gmt_files_for_mulea/tree/main/scripts_to_create_GMT_files/WikiPathways"
    click FlyAtlas "https://github.com/eltebioinformatics/gmt_files_for_mulea/tree/main/scripts_to_create_GMT_files/FlyAtlas"
    click ModEncode "https://github.com/eltebioinformatics/gmt_files_for_mulea/tree/main/scripts_to_create_GMT_files/ModEncode"
    click miRNA "https://github.com/eltebioinformatics/gmt_files_for_mulea/tree/main/scripts_to_create_GMT_files/miRNA"
    click TF "https://github.com/eltebioinformatics/gmt_files_for_mulea/tree/main/scripts_to_create_GMT_files/TF"
    click KEGG "https://github.com/eltebioinformatics/gmt_files_for_mulea/tree/main/scripts_to_create_GMT_files/KEGG"
    click Pathway_Commons "https://github.com/eltebioinformatics/gmt_files_for_mulea/tree/main/scripts_to_create_GMT_files/Pathway_Commons"
    click SignaLink "https://github.com/eltebioinformatics/gmt_files_for_mulea/tree/main/scripts_to_create_GMT_files/SignaLink"
    click Bands "https://github.com/eltebioinformatics/gmt_files_for_mulea/tree/main/scripts_to_create_GMT_files/Bands"
    click Ensembl "https://github.com/eltebioinformatics/gmt_files_for_mulea/tree/main/scripts_to_create_GMT_files/Ensembl - consecutive genes"
    click ID "https://github.com/eltebioinformatics/gmt_files_for_mulea/tree/main/scripts_to_create_GMT_files/ID_mapping_scripts"
    click DT "https://github.com/eltebioinformatics/gmt_files_for_mulea/tree/main/GMT_files/"
    click README "https://github.com/eltebioinformatics/gmt_files_for_mulea/blob/main/README.md"
    click GITIGNORE "https://github.com/eltebioinformatics/gmt_files_for_mulea/blob/main/.gitignore"

    %% Styles
    classDef external fill:#D3D3D3,stroke:#333;
    classDef etl fill:#FFA500,stroke:#333;
    classDef datastore fill:#90EE90,stroke:#333;
    classDef idmap fill:#FFFF00,stroke:#333;
    classDef consumer fill:#ADD8E6,stroke:#333;
    classDef docs fill:#FFFFFF,stroke:#333;
```
