# bartongroup/sm_varalign architecture

> Architecture diagram of the GitHub repository bartongroup/sm_varalign, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bartongroup/sm_varalign
- Repository: https://github.com/bartongroup/sm_varalign

## Overview

##### Project type and purpose
- VarAlign is an open-source, command-line bioinformatics pipeline implemented as a Python package.
- Its goal is to take a multiple sequence alignment (e.g., a Pfam alignment with human sequences), aggregate human genetic variation by column, annotate scores (e.g., conservation via AACon), and optionally perform structural analysis via ProIntVar/Arpeggio.

##### File structure
- Top-level:
- bin/ — wrapper scripts (filter_swiss, index_pfam, varalign)
- varalign/ — core Python modules and CLI entrypoint
- prointvar_config.ini, requirements.txt, environment.yml, setup.py — packaging, dependencies, configuration
- tests/ — integration/unit tests with sample data
- varalign/ subpackages:
- fetchers.py, uniprot.py, pfam.py, gnomad.py — modules pulling data from external sources
- align_variants.py, annotate_alignment.py, annotate_pfam.py — core alignment/annotation logic
- aacon.py (bundled jar), occ_gmm.py, prointvar_stats.py — scoring modules
- prointvar_analysis.py — structural branch orchestrator
- cli.py — command-line interface
- lib/aacon/ — bundled third-party conservation tool

##### Key insights from README
- Pipeline driven by CLI “varalign” command
- Uses a local config file (varalign/config.txt template) to locate VCFs, caches, external programs
- Optional branch: structural analysis with ProIntVar (Python 3) calling Arpeggio (Python 2)
- Outputs: VCFs, CSV summaries, PDF figures, and caches (.varalign, .prointvar)

##### Main components
a. User/CLI layer
- bin/varalign → varalign/cli.py
- Accepts alignment file + flags
b. Configuration manager
- varalign/config.py reading prointvar_config.ini or local config.txt
c. Data-fetcher modules
- Pfam API (index_pfam), UniProt SwissProt, Ensembl cache (SQLite), gnomAD VCF
d. Core aggregation pipeline
- Alignment parsing → annotate columns → fetch variants → align_variants
e. Scoring plugins
- Conservation: AACon (Java jar)
- Occurrence modeling: Gaussian mixture (occ_gmm)
- ProIntVar structural analysis (external program invocations via jabaws/retry)
f. File system & caches
- .varalign/ and .prointvar/ directories, SQLite caches
g. Output modules
h. External dependencies
- Java (AACon), Python 2 Arpeggio, Python 3 ProIntVar

##### Relationships & interactions
- CLI → Configuration → Orchestrator → Data fetchers → Core pipeline steps → Scoring modules
- Structural branch: after core alignment, fork into ProIntVar → Arpeggio → parse results → annotate
- All modules read/write to working directory, producing a consistent directory structure for results and cache
- Tests exercise each component in isolation (unit tests) and end-to-end (integration tests under tests/)

##### Architectural patterns & design principles
- Modular pipeline: each step implemented as a separate module with a common interface
- Plugin-style scoring: add new scoring modules by dropping a new Python file into varalign/
- Config-driven paths: avoid hardcoding, use config file priority (local vs global)
- Retry/robustness wrappers around external calls (retry.py)

##### Technologies, frameworks, libraries
- Python 3 (core code), Python 2 (Arpeggio)
- Conda environments (environment.yml)
- SQLite (built-in) for caching
- Java for AACon
- Third-party Python libraries: Biopython, numpy, pandas, click (CLI)
- External services: Pfam API, UniProt, Ensembl REST, gnomAD VEP

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Class Definitions
    classDef pipeline fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef data fill:#98FB98,stroke:#333,stroke-width:1px
    classDef external fill:#FFA500,stroke:#333,stroke-width:1px
    classDef config fill:#D3D3D3,stroke:#333,stroke-width:1px

    %% User Entrypoint
    subgraph "User Entrypoint"
        CLI_Script["bin/varalign"]:::config
        CLI_Module["varalign/cli.py"]:::config
        CLI_Script --> CLI_Module
    end

    %% Configuration
    subgraph "Configuration Manager"
        Config_Manager["varalign/config.py"]:::config
        Global_Config["prointvar_config.ini"]:::config
        Local_Config["varalign/config.txt"]:::config
        Config_Manager --> Global_Config
        Config_Manager --> Local_Config
    end
    CLI_Module --> Config_Manager

    %% Orchestrator
    Orchestrator["Pipeline Orchestrator"]:::pipeline
    Config_Manager --> Orchestrator

    %% Data Fetchers
    subgraph "Data Fetcher Modules"
        Fetchers["varalign/fetchers.py"]:::pipeline
        PfamFetcher["varalign/pfam.py"]:::pipeline
        UniProtFetcher["varalign/uniprot.py"]:::pipeline
        EnsemblFetcher["varalign/ensembl.py"]:::pipeline
        GnomadFetcher["varalign/gnomad.py"]:::pipeline
        Fetchers --> PfamFetcher
        Fetchers --> UniProtFetcher
        Fetchers --> EnsemblFetcher
        Fetchers --> GnomadFetcher
    end
    Orchestrator --> Fetchers

    %% External Services
    PfamAPI["Pfam API"]:::external
    UniProtAPI["UniProt API"]:::external
    EnsemblAPI["Ensembl REST"]:::external
    GnomadAPI["gnomAD VEP"]:::external
    PfamFetcher --> PfamAPI
    UniProtFetcher --> UniProtAPI
    EnsemblFetcher --> EnsemblAPI
    GnomadFetcher --> GnomadAPI

    %% Core Pipeline
    subgraph "Core Aggregation Pipeline"
        AlignFile["varalign/alignments.py"]:::pipeline
        AnnotateCols["varalign/annotate_alignment.py"]:::pipeline
        AnnotatePfam["varalign/annotate_pfam.py"]:::pipeline
        Mapping["varalign/mapping.py"]:::pipeline
        AlignVariants["varalign/align_variants.py"]:::pipeline
        AlignFile --> AnnotateCols
        AnnotateCols --> AnnotatePfam
        AnnotatePfam --> Mapping
        Mapping --> AlignVariants
    end
    Orchestrator --> AlignFile

    %% Scoring Plugins
    subgraph "Scoring Plugins"
        AAConWrapper["varalign/aacon.py"]:::pipeline
        AAConJar["compbio-conservation-1.1.jar"]:::external
        OccGMM["varalign/occ_gmm.py"]:::pipeline
        ProIntOrch["varalign/prointvar_analysis.py"]:::pipeline
        ProIntStats["varalign/prointvar_stats.py"]:::pipeline
        AAConWrapper --> AAConJar
        ProIntOrch --> ProIntStats
    end
    AlignVariants --> AAConWrapper
    AlignVariants --> OccGMM
    AlignVariants --> ProIntOrch

    %% External Programs
    JavaVM["Java VM"]:::external
    ProIntCLI["ProIntVar CLI"]:::external
    Arpeggio["Arpeggio (Python2)"]:::external
    AAConJar --> JavaVM
    ProIntOrch --> ProIntCLI
    ProIntOrch --> Arpeggio
    Arpeggio --> ProIntStats

    %% Wrappers & Utilities
    subgraph "External Wrappers & Utils"
        Jabaws["varalign/jabaws.py"]:::pipeline
        Retry["varalign/retry.py"]:::pipeline
        Jalview["varalign/jalview.py"]:::pipeline
        Utils["varalign/utils.py"]:::pipeline
        StatsMod["varalign/stats.py"]:::pipeline
    end
    ProIntOrch --> Retry
    AAConWrapper --> Retry
    Retry --> Jabaws
    AlignVariants --> Jalview
    Utils --> StatsMod

    %% Outputs & Caches
    subgraph "Outputs & Caches"
        VCF["VCF Files"]:::data
        CSV["CSV Summaries"]:::data
        PDF["PDF Figures"]:::data
        CacheVar[".varalign/ & SQLite Cache"]:::data
        CachePro[".prointvar/"]:::data
        AlignVariants --> VCF
        StatsMod --> CSV
        StatsMod --> PDF
        AAConWrapper --> CacheVar
        ProIntStats --> CachePro
    end

    %% Auxiliary Scripts
    subgraph "Auxiliary Commands"
        IndexPfam["bin/index_pfam"]:::config
        FilterSwiss["bin/filter_swiss"]:::config
    end

    %% Connections
    Orchestrator --> AlignFile
    OccGMM --> StatsMod
    Jabaws --> ProIntCLI

    %% Click Events
    click CLI_Script "https://github.com/bartongroup/sm_varalign/tree/master/bin/varalign"
    click CLI_Module "https://github.com/bartongroup/sm_varalign/blob/master/varalign/cli.py"
    click Config_Manager "https://github.com/bartongroup/sm_varalign/blob/master/varalign/config.py"
    click Global_Config "https://github.com/bartongroup/sm_varalign/blob/master/prointvar_config.ini"
    click Local_Config "https://github.com/bartongroup/sm_varalign/blob/master/varalign/config.txt"
    click Fetchers "https://github.com/bartongroup/sm_varalign/blob/master/varalign/fetchers.py"
    click PfamFetcher "https://github.com/bartongroup/sm_varalign/blob/master/varalign/pfam.py"
    click UniProtFetcher "https://github.com/bartongroup/sm_varalign/blob/master/varalign/uniprot.py"
    click EnsemblFetcher "https://github.com/bartongroup/sm_varalign/blob/master/varalign/ensembl.py"
    click GnomadFetcher "https://github.com/bartongroup/sm_varalign/blob/master/varalign/gnomad.py"
    click AlignFile "https://github.com/bartongroup/sm_varalign/blob/master/varalign/alignments.py"
    click AnnotateCols "https://github.com/bartongroup/sm_varalign/blob/master/varalign/annotate_alignment.py"
    click AnnotatePfam "https://github.com/bartongroup/sm_varalign/blob/master/varalign/annotate_pfam.py"
    click Mapping "https://github.com/bartongroup/sm_varalign/blob/master/varalign/mapping.py"
    click AlignVariants "https://github.com/bartongroup/sm_varalign/blob/master/varalign/align_variants.py"
    click AAConWrapper "https://github.com/bartongroup/sm_varalign/blob/master/varalign/aacon.py"
    click AAConJar "https://github.com/bartongroup/sm_varalign/blob/master/varalign/lib/aacon/compbio-conservation-1.1.jar"
    click OccGMM "https://github.com/bartongroup/sm_varalign/blob/master/varalign/occ_gmm.py"
    click ProIntOrch "https://github.com/bartongroup/sm_varalign/blob/master/varalign/prointvar_analysis.py"
    click ProIntStats "https://github.com/bartongroup/sm_varalign/blob/master/varalign/prointvar_stats.py"
    click Jabaws "https://github.com/bartongroup/sm_varalign/blob/master/varalign/jabaws.py"
    click Retry "https://github.com/bartongroup/sm_varalign/blob/master/varalign/retry.py"
    click Jalview "https://github.com/bartongroup/sm_varalign/blob/master/varalign/jalview.py"
    click Utils "https://github.com/bartongroup/sm_varalign/blob/master/varalign/utils.py"
    click StatsMod "https://github.com/bartongroup/sm_varalign/blob/master/varalign/stats.py"
    click IndexPfam "https://github.com/bartongroup/sm_varalign/tree/master/bin/index_pfam"
    click FilterSwiss "https://github.com/bartongroup/sm_varalign/tree/master/bin/filter_swiss"
```
