How helloworldlty/ukbiolm works
Project Type and Purpose
• This is a genomics-focused deep learning research codebase. It provides:
• A core Python library (grelu) for data I/O, preprocessing, model definitions, interpretation, and visualization.
• A pre-training pipeline using large variant-augmented genomic datasets.
• Fine-tuning and inference applications for cell-line improvement and variant effect prediction (UKBioBERT, UKBioFormer).
• Notebooks and scripts for data preparation, pre-training, and downstream applications.
File Structure
• Top-level packages:
• grelu/: Core reusable modules:
– data/, transforms/: dataset loading, augmentation, preprocessing.
– model/, lightning/: model architectures, heads, losses, training utilities using PyTorch Lightning.
– io/: readers/writers for FASTA, BED, BigWig, MEME, genome resources.
– resources/: static blacklists and motif libraries.
– sequence/, utils.py, design.py, variant.py: utility functions.
• pretraining/: Jupyter notebooks and scripts (run_mlm.py, run_pretrain.sh, etc.) to prepare datasets and run masked-LM or contrastive pre-training over UK Biobank, GTEx, ROSMAP.
• improve_cellline/: scripts and models for fine-tuning EPInformer on cell-line data.
• ukbiobert_application/ and ukbioformer_application/: notebooks and config files for downstream inference and visualization using the pre-trained models.
• Environment YAMLs (ukbiobert_pretrain.yml, etc.): specify dependencies and GPU configurations.
Extract Key Architecture Details from README
• Environments are separated by model type (UKBioBERT vs. UKBioFormer) and hardware (H100 vs. no-H100).
• Data sources: UK Biobank, GTEx, ROSMAP (external databases).
• Pre-training → model weights → fine-tuning/inference → downstream analyses.
• External frameworks: Huggingface, PyTorch Lightning, grelu base library.
Main Components & Interactions
a. Core Library (grelu):
• Data ingestion & preprocessing
• Model building blocks & trunks (Enformer, Borzoi, ExplainN)
• Training utilities (losses, metrics via Lightning)
• Interpretation & visualization tools
b. Data Sources (External):
• UK Biobank, GTEx, ROSMAP raw genomic data
• Pre-computed reference sequences and variant catalogs
c. Pre-training Pipeline:
• Data pipeline (variant + reference → masked LM samples)
• Model instantiation from grelu.model.trunks
• Training loop via PyTorch Lightning
• Checkpoint saving
d. Fine-tuning / Application Pipelines:
• improve_cellline: EPInformer model training on cell-line data
• ukbiobert_application / ukbioformer_application: notebooks for inference, ensemble, variant effect checks
• Use of pre-trained checkpoints, further training/inference
e. Environments & Infrastructure:
• Conda environments for each pipeline/hardware setup
• GPU clusters (H100, A100, A40)
• Shell scripts (run_pretrain.sh, download_data.sh) for orchestration
Architectural Patterns & Design Principles
• Modular core library (grelu) that separates concerns: I/O, data transforms, model definitions, training, and interpretation.
• Pipeline orchestration via standalone scripts/notebooks.
• Use of PyTorch Lightning for standardized training loops.
• Clear separation of pre-training vs. fine-tuning workflows.
• Externalization of static resources (blacklists, motif files) under grelu/resources.
Final Checklist
• Clear labels for each component and sub‐component.
• External dependencies and resource contexts.