How opentargets/gentropy works
Main Components
a) Core Processing Layer:
• Spark Processing Engine (evidenced by spark_helpers.py and session.py)
• Data Processing Pipeline Components
• Method Implementation Layer
b) Data Management Layer:
• Dataset Management System
• Data Source Connectors
• Schema Management
c) Analysis Components:
• Statistical Analysis Tools
• Fine Mapping Systems
• Colocalization Analysis
• L2G (Likely to Gene) Prediction System
Component Relationships
Input Sources → Data Ingestion → Processing → Analysis → Output
With parallel flows for:
• Schema Validation
• Quality Control
• Data Harmonization
Detailed Component Breakdown
a) Data Sources (show as external systems):
• GWAS Catalog
• EQTl Catalogue
• FinnGen
• GnomAD
• UK Biobank
• Open Targets
b) Core Processing Components (show as internal modules):
• Dataset Management (/dataset)
• Method Implementation (/method)
• Common Utilities (/common)
• Data Source Connectors (/datasource)
c) Analysis Methods (show as processing modules):
• CARMA
• PICS
• SuSiE
• Colocalization
• L2G Prediction
Technical Implementation Details
• Rectangle: Core processing modules
• Hexagon: External data sources
• Diamond: Analysis methods
• Blue: Core components
• Green: Data sources
• Orange: Analysis methods
• Gray: External systems
• Purple: Quality control and validation
Key Architectural Patterns to Highlight
• Separation between data ingestion, processing, and analysis
• Modular nature of the analysis methods
• Central role of the Spark processing engine
• Schema validation flow
• Quality control checkpoints
Additional Elements
Annotations for:
• Data formats (Parquet, JSON, TSV)
• Key technologies (Apache Spark, Python)
• Validation points
• Configuration management