How compepigen/methylbert works
Project type and purpose
• This is a domain-specific Python package and command-line tool (“MethylBERT”) for read-level DNA methylation pattern classification and tumour deconvolution.
• It wraps PyTorch/Transformers under the hood, providing three main CLI commands: preprocess_finetune, finetune, and deconvolute.
File structure
• Top-level:
• Dockerfile, environment.yml, requirements.txt, pyproject.toml – container and build configurations
• config/ – JSON files defining default parameters for each pipeline stage
• src/methylbert/ – core library modules
• test/ – unit tests
• tutorials/ – usage examples
• src/methylbert/ substructure:
• cli.py – CLI entry point & argument parsing
• config.py – loads JSON configs
• data/ – data-loading and preprocessing (BAM reading, genome, k-mers, dataset I/O)
• network.py – Transformer model definition (based on HuggingFace)
• trainer.py – training loop, metric logging, checkpointing
• deconvolute.py – tumour deconvolution inference logic
• utils.py/function.py – shared utilities
README for additional insights
• Three distinct pipeline stages driven by the CLI.
• External dependencies: PyTorch, HuggingFace Transformers, condaforge packages, CUDA.
• Input/outputs are strictly file-based (BAM, CSV, FASTA, model dirs, result dirs).
Main components and interactions
a. Components
1. CLI Layer (src/methylbert/cli.py)
2. Configuration Layer (config/*.json + config.py)
3. Data Layer (src/methylbert/data/*.py)
4. Model Layer (src/methylbert/network.py)
5. Training Orchestrator (src/methylbert/trainer.py)
6. Inference Layer (src/methylbert/deconvolute.py)
7. Utilities (src/methylbert/utils.py, function.py)
8. Tests & Tutorials
9. External Dependencies (PyTorch, Transformers, CUDA, file I/O)
b. Relationships & data flow
• CLI parses user arguments → loads JSON configs → invokes one of three pipelines
• Preprocess pipeline: Data Layer reads BAM/DMR/FASTA → generates train/test CSV
• Finetune pipeline: Data Layer loads CSV → Trainer pulls batches → Model Layer (Transformer) → backprop via PyTorch → checkpoints saved
• Deconvolute pipeline: Data Layer reads bulk data → loads trained model → Deconvolute logic → outputs purity estimates
• Utilities and config are shared across all stages
c. Architectural patterns
• Modular separation: CLI / config / data / model / training / inference
• Pipeline pattern: sequential stages with file-based hand-off
• Library + CLI wrapper pattern for re-use and scripting
d. Key technologies
• Python packaging (pyproject.toml, pip, conda)
• PyTorch & HuggingFace Transformers
• Multiprocessing for data preprocessing
• JSON/YAML for configuration
• Docker for containerization
Detailed elements
• CLI entrypoint and three sub-commands
• Config loader with JSON files
• Data ingestion modules (bam.py, genome.py, dataset.py)
• K-mer vocabulary generator (vocab.py)
• Transformer model definition and its reliance on HuggingFace
• Trainer loop with backprop, checkpointing, metrics logging
• Deconvolution algorithm module, batch inference, post-processing
• Test suite (optional as a separate validation component)
• Tutorials folder as documentation reference