How bartongroup/sm_varalign works
Project type and purpose
• VarAlign is an open-source, command-line bioinformatics pipeline implemented as a Python package.
• Its goal is to take a multiple sequence alignment (e.g., a Pfam alignment with human sequences), aggregate human genetic variation by column, annotate scores (e.g., conservation via AACon), and optionally perform structural analysis via ProIntVar/Arpeggio.
File structure
• Top-level:
• bin/ — wrapper scripts (filter_swiss, index_pfam, varalign)
• varalign/ — core Python modules and CLI entrypoint
• prointvar_config.ini, requirements.txt, environment.yml, setup.py — packaging, dependencies, configuration
• tests/ — integration/unit tests with sample data
• varalign/ subpackages:
• fetchers.py, uniprot.py, pfam.py, gnomad.py — modules pulling data from external sources
• align_variants.py, annotate_alignment.py, annotate_pfam.py — core alignment/annotation logic
• aacon.py (bundled jar), occ_gmm.py, prointvar_stats.py — scoring modules
• prointvar_analysis.py — structural branch orchestrator
• cli.py — command-line interface
• lib/aacon/ — bundled third-party conservation tool
Key insights from README
• Pipeline driven by CLI “varalign” command
• Uses a local config file (varalign/config.txt template) to locate VCFs, caches, external programs
• Optional branch: structural analysis with ProIntVar (Python 3) calling Arpeggio (Python 2)
• Outputs: VCFs, CSV summaries, PDF figures, and caches (.varalign, .prointvar)
Main components
a. User/CLI layer
• bin/varalign → varalign/cli.py
• Accepts alignment file + flags
b. Configuration manager
• varalign/config.py reading prointvar_config.ini or local config.txt
c. Data-fetcher modules
• Pfam API (index_pfam), UniProt SwissProt, Ensembl cache (SQLite), gnomAD VCF
d. Core aggregation pipeline
• Alignment parsing → annotate columns → fetch variants → align_variants
e. Scoring plugins
• Conservation: AACon (Java jar)
• Occurrence modeling: Gaussian mixture (occ_gmm)
• ProIntVar structural analysis (external program invocations via jabaws/retry)
f. File system & caches
• .varalign/ and .prointvar/ directories, SQLite caches
g. Output modules
h. External dependencies
• Java (AACon), Python 2 Arpeggio, Python 3 ProIntVar
Relationships & interactions
• CLI → Configuration → Orchestrator → Data fetchers → Core pipeline steps → Scoring modules
• Structural branch: after core alignment, fork into ProIntVar → Arpeggio → parse results → annotate
• All modules read/write to working directory, producing a consistent directory structure for results and cache
• Tests exercise each component in isolation (unit tests) and end-to-end (integration tests under tests/)
Architectural patterns & design principles
• Modular pipeline: each step implemented as a separate module with a common interface
• Plugin-style scoring: add new scoring modules by dropping a new Python file into varalign/
• Config-driven paths: avoid hardcoding, use config file priority (local vs global)
• Retry/robustness wrappers around external calls (retry.py)
Technologies, frameworks, libraries
• Python 3 (core code), Python 2 (Arpeggio)
• Conda environments (environment.yml)
• SQLite (built-in) for caching
• Java for AACon
• Third-party Python libraries: Biopython, numpy, pandas, click (CLI)
• External services: Pfam API, UniProt, Ensembl REST, gnomAD VEP