# opentargets/gentropy architecture

> Architecture diagram of the GitHub repository opentargets/gentropy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/opentargets/gentropy
- Repository: https://github.com/opentargets/gentropy

## Overview

##### Main Components

a) Core Processing Layer:
- Spark Processing Engine (evidenced by spark_helpers.py and session.py)
- Data Processing Pipeline Components
- Method Implementation Layer

b) Data Management Layer:
- Dataset Management System
- Data Source Connectors
- Schema Management

c) Analysis Components:
- Statistical Analysis Tools
- Fine Mapping Systems
- Colocalization Analysis
- L2G (Likely to Gene) Prediction System

##### Component Relationships

**Input Sources → Data Ingestion → Processing → Analysis → Output**

**With parallel flows for:**
- Schema Validation
- Quality Control
- Data Harmonization

##### Detailed Component Breakdown

a) Data Sources (show as external systems):
- GWAS Catalog
- EQTl Catalogue
- FinnGen
- GnomAD
- UK Biobank
- Open Targets

b) Core Processing Components (show as internal modules):
- Dataset Management (/dataset)
- Method Implementation (/method)
- Common Utilities (/common)
- Data Source Connectors (/datasource)

c) Analysis Methods (show as processing modules):
- CARMA
- PICS
- SuSiE
- Colocalization
- L2G Prediction

##### Technical Implementation Details

- Rectangle: Core processing modules
- Hexagon: External data sources
- Diamond: Analysis methods

- Blue: Core components
- Green: Data sources
- Orange: Analysis methods
- Gray: External systems
- Purple: Quality control and validation

##### Key Architectural Patterns to Highlight

- Separation between data ingestion, processing, and analysis
- Modular nature of the analysis methods
- Central role of the Spark processing engine
- Schema validation flow
- Quality control checkpoints

##### Additional Elements

Annotations for:
- Data formats (Parquet, JSON, TSV)
- Key technologies (Apache Spark, Python)
- Validation points
- Configuration management

## Diagram (Mermaid)

```mermaid
graph TB
    %% Styles
    classDef external fill:#gray,stroke:#333,stroke-width:2px
    classDef core fill:#blue,stroke:#333,stroke-width:2px
    classDef analysis fill:#orange,stroke:#333,stroke-width:2px
    classDef data fill:#green,stroke:#333,stroke-width:2px
    classDef validation fill:#purple,stroke:#333,stroke-width:2px

    %% External Data Sources
    subgraph DataSources["External Data Sources"]
        GWAS["GWAS Catalog"]:::external
        EQTL["EQTl Catalogue"]:::external
        FINN["FinnGen"]:::external
        GNOM["GnomAD"]:::external
        UKBB["UK Biobank"]:::external
        OT["Open Targets"]:::external
    end

    %% Core Processing Layer
    subgraph CoreProcessing["Core Processing Layer"]
        SPARK["Spark Processing Engine"]:::core
        DMS["Dataset Management System"]:::core
        SCHEMA["Schema Management"]:::core
        
        subgraph DataConnectors["Data Source Connectors"]
            DC_GWAS["GWAS Connector"]:::core
            DC_EQTL["EQTL Connector"]:::core
            DC_FINN["FinnGen Connector"]:::core
            DC_GNOM["GnomAD Connector"]:::core
            DC_UKBB["UKBB Connector"]:::core
            DC_OT["OT Connector"]:::core
        end
    end

    %% Analysis Layer
    subgraph Analysis["Analysis Methods"]
        CARMA["CARMA"]:::analysis
        PICS["PICS"]:::analysis
        SUSIE["SuSiE"]:::analysis
        COLOC["Colocalization"]:::analysis
        L2G["L2G Prediction"]:::analysis
    end

    %% Relationships
    GWAS --> DC_GWAS
    EQTL --> DC_EQTL
    FINN --> DC_FINN
    GNOM --> DC_GNOM
    UKBB --> DC_UKBB
    OT --> DC_OT

    DC_GWAS & DC_EQTL & DC_FINN & DC_GNOM & DC_UKBB & DC_OT --> SPARK
    SPARK --> DMS
    DMS --> SCHEMA
    SCHEMA --> Analysis

    %% Click Events
    click SPARK "https://github.com/opentargets/gentropy/blob/dev/src/gentropy/common/session.py"
    click DMS "https://github.com/opentargets/gentropy/blob/dev/src/gentropy/dataset/dataset.py"
    click SCHEMA "https://github.com/opentargets/gentropy/tree/dev/src/gentropy/assets/schemas/"
    click CARMA "https://github.com/opentargets/gentropy/blob/dev/src/gentropy/method/carma.py"
    click PICS "https://github.com/opentargets/gentropy/blob/dev/src/gentropy/method/pics.py"
    click SUSIE "https://github.com/opentargets/gentropy/blob/dev/src/gentropy/method/susie_inf.py"
    click COLOC "https://github.com/opentargets/gentropy/blob/dev/src/gentropy/method/colocalisation.py"
    click L2G "https://github.com/opentargets/gentropy/tree/dev/src/gentropy/method/l2g/"
    click DC_GWAS "https://github.com/opentargets/gentropy/tree/dev/src/gentropy/datasource/gwas_catalog/"
    click DC_EQTL "https://github.com/opentargets/gentropy/tree/dev/src/gentropy/datasource/eqtl_catalogue/"
    click DC_FINN "https://github.com/opentargets/gentropy/tree/dev/src/gentropy/datasource/finngen/"
    click DC_GNOM "https://github.com/opentargets/gentropy/tree/dev/src/gentropy/datasource/gnomad/"
    click DC_UKBB "https://github.com/opentargets/gentropy/tree/dev/src/gentropy/datasource/ukbiobank/"
    click DC_OT "https://github.com/opentargets/gentropy/tree/dev/src/gentropy/datasource/open_targets/"

    %% Legend
    subgraph Legend
        L1["External Systems"]:::external
        L2["Core Components"]:::core
        L3["Analysis Methods"]:::analysis
        L4["Data Sources"]:::data
        L5["Validation"]:::validation
    end
```
