# cloufield/gwaslab architecture

> Architecture diagram of the GitHub repository cloufield/gwaslab, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cloufield/gwaslab
- Repository: https://github.com/cloufield/gwaslab

## Overview

- Data Input & Preprocessing Module:
- Components: Data loaders and input format parsers (see files like g_Sumstats.py, io_preformat_input.py, io_read_tabular.py).
- Function: Ingest GWAS sumstats and format them for further processing.
- Standardization & Normalization Engine:
- Components: Variant ID standardizer, CHR/POS normalization (e.g., g_headers.py, qc_fix_sumstats.py, hm_casting.py).
- Function: Ensure consistency in columns and variant representations.
- Harmonization Module:
- Components: rsID assignment, CHR/POS assignment, allele alignment (files such as hm_harmonize_sumstats.py, hm_rsid_to_chrpos.py).
- Function: Align and correct allele information with reference data.
- Quality Control & Data Filtering:
- Components: Quality checks, filtering utilities, extreme value removal (e.g., qc_check_datatype.py, util_in_filter_value.py).
- Function: Validate and clean the GWAS data.
- Statistical Conversion & Calculation:
- Components: Conversions between statistics (BETA/SE, OR conversions etc.) and calculations (g_Sumstats_summary.py, util_in_calculate_gc.py).
- Function: Compute summary metrics required for further analysis.
- Visualization Suite:
- Components: Multiple plotting modules for Manhattan, QQ/MQQ, Miami, Brisbane, and other exploratory visuals (e.g., viz_plot_miamiplot.py, viz_plot_trumpetplot.py).
- Function: Render the processed data into publication-quality plots.
- Utility & Integration Tools:
- Components: Logging, file caching, command-line integration (see g_Log.py, run_script.py, cache_manager.py).
- Function: Support operations by maintaining logs, caching intermediate results, and enabling reproducible workflows.
- External Interfaces and Deployment:
- Components: Docker deployment (docker/Dockerfile), conda environment definitions (environment.yml files), and examples demonstrating the API usage.
- Function: Facilitate installation, environment setup, and integration into various computational pipelines.

##### Relationships and Data Flow
- Then, let data flow to the Harmonization Module where alleles and variant positions are corrected or realigned using reference data.
- Follow this with the Quality Control & Filtering step ensuring only “clean” data progresses.
- From here, branch into two streams:
- One stream directs data into Statistical Conversions/Calculations (e.g., converting to LD scores, calculating power, etc.).
- The other stream feeds into the Visualization Suite to produce several types of figures and plots.
- A side component for Utility & Integration that cuts across these phases (e.g., logging every transformation, caching results, and providing command-line access).
- Finally, include deployment artifacts (Docker, conda environments) that help package and deliver the solution.

##### Architectural Patterns and Design Principles
- Modular and Object-Oriented: Most functionalities are encapsulated as separate modules and methods (e.g., the Sumstats object that ties multiple processes together).
- Extensibility: The design supports easy swapping or addition of new transformation methods as each function is self-contained.
- Separation of Concerns: The clear split between input handling, data processing, and visualization simplifies maintenance and updates.

##### Final Advice
- Grouping related functions (for instance, all visualization functions under one umbrella with sub-nodes for each plot type).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Input & Preprocessing
    subgraph "Data Input & Preprocessing"
        DI1["Data Loader (g_Sumstats)"]:::data_module
        DI2["Input Formatter (io_preformat_input)"]:::data_module
        DI3["Tabular Reader (io_read_tabular)"]:::data_module
        DI1 --> DI2 --> DI3
    end

    %% Standardization & Normalization Engine
    subgraph "Standardization & Normalization"
        SN1["Header Standardizer (g_headers)"]:::standard_module
        SN2["Fix Sumstats (qc_fix_sumstats)"]:::standard_module
        SN3["Casting Utility (hm_casting)"]:::standard_module
        SN1 --> SN2 --> SN3
    end

    %% Harmonization Module
    subgraph "Harmonization Module"
        HM1["Harmonize Sumstats (hm_harmonize_sumstats)"]:::harmonization_module
        HM2["rsID to CHR/POS (hm_rsid_to_chrpos)"]:::harmonization_module
        HM1 --> HM2
    end

    %% Quality Control & Data Filtering
    subgraph "Quality Control & Filtering"
        QC1["Datatype Check (qc_check_datatype)"]:::qc_module
        QC2["Filter Utility (util_in_filter_value)"]:::qc_module
        QC1 --> QC2
    end

    %% Statistical Conversion & Calculation
    subgraph "Statistical Conversion"
        SC1["Sumstats Summary (g_Sumstats_summary)"]:::conversion_module
        SC2["Calc GC Utility (util_in_calculate_gc)"]:::conversion_module
        SC1 --> SC2
    end

    %% Visualization Suite
    subgraph "Visualization Suite"
        VS1["Miami Plot (viz_plot_miamiplot)"]:::viz_module
        VS2["Trumpet Plot (viz_plot_trumpetplot)"]:::viz_module
        VS3["MQQ Plot (viz_plot_mqqplot)"]:::viz_module
        VS4["Other Plot Modules (viz_plot_*)"]:::viz_module
        VS1 --> VS2 --> VS3 --> VS4
    end

    %% Utility & Integration Tools
    subgraph "Utility & Integration Tools"
        UT1["Logging (g_Log)"]:::utility_module
        UT2["Script Runner (run_script)"]:::utility_module
        UT3["Cache Manager (cache_manager)"]:::utility_module
        UT1 --> UT2 --> UT3
    end

    %% External Interfaces & Deployment
    subgraph "External Interfaces & Deployment"
        EI1["Dockerfile (Dockerfile)"]:::external_module
        EI2["Conda Env (environment.yml)"]:::external_module
        EI3["Conda Env 3.4.40 (environment_3.4.40.yml)"]:::external_module
        EI1 --> EI2 --> EI3
    end

    %% Pipeline Flow Connections
    DI3 --> SN1
    SN3 --> HM1
    HM2 --> QC1
    QC2 --> SC1
    QC2 --> VS1
    SC2 --> EI1
    VS4 --> EI1

    %% Utility dotted connections (side integration)
    DI2 -.-> UT1
    SN2 -.-> UT1
    HM2 -.-> UT1
    QC2 -.-> UT1
    SC2 -.-> UT1
    VS4 -.-> UT1

    %% Utility to External Interfaces connection
    UT1 -.-> EI1

    %% Class Definitions for Colors
    classDef data_module fill:#f9c74f,stroke:#333,stroke-width:2px;
    classDef standard_module fill:#90be6d,stroke:#333,stroke-width:2px;
    classDef harmonization_module fill:#f9844a,stroke:#333,stroke-width:2px;
    classDef qc_module fill:#f94144,stroke:#333,stroke-width:2px;
    classDef conversion_module fill:#577590,stroke:#333,stroke-width:2px;
    classDef viz_module fill:#277da1,stroke:#333,stroke-width:2px;
    classDef utility_module fill:#9b5de5,stroke:#333,stroke-width:2px;
    classDef external_module fill:#f3722c,stroke:#333,stroke-width:2px;

    %% Click Events for Data Input & Preprocessing
    click DI1 "https://github.com/cloufield/gwaslab/blob/main/src/gwaslab/g_Sumstats.py"
    click DI2 "https://github.com/cloufield/gwaslab/blob/main/src/gwaslab/io_preformat_input.py"
    click DI3 "https://github.com/cloufield/gwaslab/blob/main/src/gwaslab/io_read_tabular.py"

    %% Click Events for Standardization & Normalization Engine
    click SN1 "https://github.com/cloufield/gwaslab/blob/main/src/gwaslab/g_headers.py"
    click SN2 "https://github.com/cloufield/gwaslab/blob/main/src/gwaslab/qc_fix_sumstats.py"
    click SN3 "https://github.com/cloufield/gwaslab/blob/main/src/gwaslab/hm_casting.py"

    %% Click Events for Harmonization Module
    click HM1 "https://github.com/cloufield/gwaslab/blob/main/src/gwaslab/hm_harmonize_sumstats.py"
    click HM2 "https://github.com/cloufield/gwaslab/blob/main/src/gwaslab/hm_rsid_to_chrpos.py"

    %% Click Events for Quality Control & Data Filtering
    click QC1 "https://github.com/cloufield/gwaslab/blob/main/src/gwaslab/qc_check_datatype.py"
    click QC2 "https://github.com/cloufield/gwaslab/blob/main/src/gwaslab/util_in_filter_value.py"

    %% Click Events for Statistical Conversion & Calculation
    click SC1 "https://github.com/cloufield/gwaslab/blob/main/src/gwaslab/g_Sumstats_summary.py"
    click SC2 "https://github.com/cloufield/gwaslab/blob/main/src/gwaslab/util_in_calculate_gc.py"

    %% Click Events for Visualization Suite
    click VS1 "https://github.com/cloufield/gwaslab/blob/main/src/gwaslab/viz_plot_miamiplot.py"
    click VS2 "https://github.com/cloufield/gwaslab/blob/main/src/gwaslab/viz_plot_trumpetplot.py"
    click VS3 "https://github.com/cloufield/gwaslab/blob/main/src/gwaslab/viz_plot_mqqplot.py"
    click VS4 "https://github.com/cloufield/gwaslab/tree/main/src/gwaslab"

    %% Click Events for Utility & Integration Tools
    click UT1 "https://github.com/cloufield/gwaslab/blob/main/src/gwaslab/g_Log.py"
    click UT2 "https://github.com/cloufield/gwaslab/blob/main/src/gwaslab/run_script.py"
    click UT3 "https://github.com/cloufield/gwaslab/blob/main/src/gwaslab/cache_manager.py"

    %% Click Events for External Interfaces & Deployment
    click EI1 "https://github.com/cloufield/gwaslab/tree/main/docker/Dockerfile"
    click EI2 "https://github.com/cloufield/gwaslab/blob/main/environment.yml"
    click EI3 "https://github.com/cloufield/gwaslab/blob/main/environment_3.4.40.yml"
```
