# compepigen/methylbert architecture

> Architecture diagram of the GitHub repository compepigen/methylbert, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/compepigen/methylbert
- Repository: https://github.com/compepigen/methylbert

## Overview

##### Project type and purpose
- This is a domain-specific Python package and command-line tool (“MethylBERT”) for read-level DNA methylation pattern classification and tumour deconvolution.
- It wraps PyTorch/Transformers under the hood, providing three main CLI commands: preprocess_finetune, finetune, and deconvolute.

##### File structure
- Top-level:
- Dockerfile, environment.yml, requirements.txt, pyproject.toml – container and build configurations
- config/ – JSON files defining default parameters for each pipeline stage
- src/methylbert/ – core library modules
- test/ – unit tests
- tutorials/ – usage examples
- src/methylbert/ substructure:
- cli.py – CLI entry point & argument parsing
- config.py – loads JSON configs
- data/ – data-loading and preprocessing (BAM reading, genome, k-mers, dataset I/O)
- network.py – Transformer model definition (based on HuggingFace)
- trainer.py – training loop, metric logging, checkpointing
- deconvolute.py – tumour deconvolution inference logic
- utils.py/function.py – shared utilities

##### README for additional insights
- Three distinct pipeline stages driven by the CLI.
- External dependencies: PyTorch, HuggingFace Transformers, condaforge packages, CUDA.
- Input/outputs are strictly file-based (BAM, CSV, FASTA, model dirs, result dirs).

##### Main components and interactions
a. Components
1. CLI Layer (src/methylbert/cli.py)
2. Configuration Layer (config/*.json + config.py)
3. Data Layer (src/methylbert/data/*.py)
4. Model Layer (src/methylbert/network.py)
5. Training Orchestrator (src/methylbert/trainer.py)
6. Inference Layer (src/methylbert/deconvolute.py)
7. Utilities (src/methylbert/utils.py, function.py)
8. Tests & Tutorials
9. External Dependencies (PyTorch, Transformers, CUDA, file I/O)
b. Relationships & data flow
- CLI parses user arguments → loads JSON configs → invokes one of three pipelines
- Preprocess pipeline: Data Layer reads BAM/DMR/FASTA → generates train/test CSV
- Finetune pipeline: Data Layer loads CSV → Trainer pulls batches → Model Layer (Transformer) → backprop via PyTorch → checkpoints saved
- Deconvolute pipeline: Data Layer reads bulk data → loads trained model → Deconvolute logic → outputs purity estimates
- Utilities and config are shared across all stages
c. Architectural patterns
- Modular separation: CLI / config / data / model / training / inference
- Pipeline pattern: sequential stages with file-based hand-off
- Library + CLI wrapper pattern for re-use and scripting
d. Key technologies
- Python packaging (pyproject.toml, pip, conda)
- PyTorch & HuggingFace Transformers
- Multiprocessing for data preprocessing
- JSON/YAML for configuration
- Docker for containerization

##### Detailed elements
- CLI entrypoint and three sub-commands
- Config loader with JSON files
- Data ingestion modules (bam.py, genome.py, dataset.py)
- K-mer vocabulary generator (vocab.py)
- Transformer model definition and its reliance on HuggingFace
- Trainer loop with backprop, checkpointing, metrics logging
- Deconvolution algorithm module, batch inference, post-processing
- Test suite (optional as a separate validation component)
- Tutorials folder as documentation reference

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI & Config Layer
    subgraph "CLI & Configuration"
        direction TB
        CLI["CLI Entry Point\n(cli.py)"]:::cli
        ConfigLoader["Config Loader\n(config.py)"]:::config
        JSON1["preprocess_finetune_config.json"]:::config
        JSON2["finetune_config.json"]:::config
        JSON3["deconvolute_config.json"]:::config
    end

    CLI -->|"parse args"| ConfigLoader
    ConfigLoader -->|"load JSON"| JSON1
    ConfigLoader -->|"load JSON"| JSON2
    ConfigLoader -->|"load JSON"| JSON3

    %% Preprocess Pipeline
    subgraph "Preprocess Pipeline" 
        direction TB
        DataBAM["BAM Reader\n(bam.py)"]:::data
        Genome["Genome Loader\n(genome.py)"]:::data
        Vocab["K-mer Generator\n(vocab.py)"]:::data
        PreGen["Data Generator\n(finetune_data_generate.py)"]:::data
        CSV["train_seq.csv / test_seq.csv"]:::io
    end

    ConfigLoader -->|"preprocess_finetune"| DataBAM
    DataBAM -->|"reads BAM"| Genome
    Genome --> Vocab
    Vocab --> PreGen
    PreGen -->|"outputs CSV"| CSV

    %% Finetune Pipeline
    subgraph "Finetune Pipeline" 
        direction TB
        Dataset["Dataset Loader\n(dataset.py)"]:::data
        Trainer["Training Orchestrator\n(trainer.py)"]:::model
        ModelDef["Transformer Model\n(network.py)"]:::model
        Checkpoint[".pt Model Checkpoint"]:::io
    end

    ConfigLoader -->|"finetune"| Dataset
    CSV -->|"load CSV"| Dataset
    Dataset -->|"batches"| Trainer
    Trainer -->|"forward/backprop"| ModelDef
    ModelDef -->|"save checkpoint"| Checkpoint

    %% Deconvolution Pipeline
    subgraph "Deconvolution Pipeline"
        direction TB
        BulkData["Bulk Data Loader\n(dataset.py)"]:::data
        Deconv["Deconvolution Logic\n(deconvolute.py)"]:::inference
        Purity["purity.csv"]:::io
    end

    ConfigLoader -->|"deconvolute"| BulkData
    BulkData -->|"load bulk"| Deconv
    Checkpoint -->|"load model"| Deconv
    Deconv -->|"output purity"| Purity

    %% Utilities
    subgraph "Utilities"
        direction TB
        Utils["Utils Module\n(utils.py)"]:::util
        Funcs["Functions\n(function.py)"]:::util
    end

    CLI --> Utils
    CLI --> Funcs
    ConfigLoader --> Utils
    ConfigLoader --> Funcs
    Dataset --> Utils
    Deconv --> Utils

    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        PyTorch["PyTorch"]:::external
        HF["HuggingFace Transformers"]:::external
        CUDA["CUDA"]:::external
        FileIO["File System I/O"]:::external
    end

    ModelDef --> PyTorch
    ModelDef --> HF
    Trainer --> PyTorch
    DataBAM --> FileIO
    Genome --> FileIO
    PreGen --> FileIO
    BulkData --> FileIO

    %% Testing & Tutorials
    subgraph "Tests"
        direction TB
        TestDeconv["test_deconvolute.py"]:::test
        TestFT["test_finetune.py"]:::test
        TestPre["test_finetune_preprocess.py"]:::test
        TestData["test/data/*"]:::test
    end

    subgraph "Tutorials"
        direction TB
        T1["01_Data_Preparation.md"]:::tutorial
        T2["02_Preprocessing_training_data.ipynb"]:::tutorial
        T3["03_Preprocessing_bulk_data.ipynb"]:::tutorial
        T4["04_Fine-tuning_MethylBERT_model.ipynb"]:::tutorial
        T5["05_tumour_deconvolution.ipynb"]:::tutorial
    end

    %% Click Events
    click CLI "https://github.com/compepigen/methylbert/blob/main/src/methylbert/cli.py"
    click ConfigLoader "https://github.com/compepigen/methylbert/blob/main/src/methylbert/config.py"
    click JSON1 "https://github.com/compepigen/methylbert/blob/main/config/preprocess_finetune_config.json"
    click JSON2 "https://github.com/compepigen/methylbert/blob/main/config/finetune_config.json"
    click JSON3 "https://github.com/compepigen/methylbert/blob/main/config/deconvolute_config.json"
    click DataBAM "https://github.com/compepigen/methylbert/blob/main/src/methylbert/data/bam.py"
    click Genome "https://github.com/compepigen/methylbert/blob/main/src/methylbert/data/genome.py"
    click Vocab "https://github.com/compepigen/methylbert/blob/main/src/methylbert/data/vocab.py"
    click PreGen "https://github.com/compepigen/methylbert/blob/main/src/methylbert/data/finetune_data_generate.py"
    click Dataset "https://github.com/compepigen/methylbert/blob/main/src/methylbert/data/dataset.py"
    click ModelDef "https://github.com/compepigen/methylbert/blob/main/src/methylbert/network.py"
    click Trainer "https://github.com/compepigen/methylbert/blob/main/src/methylbert/trainer.py"
    click Deconv "https://github.com/compepigen/methylbert/blob/main/src/methylbert/deconvolute.py"
    click Utils "https://github.com/compepigen/methylbert/blob/main/src/methylbert/utils.py"
    click Funcs "https://github.com/compepigen/methylbert/blob/main/src/methylbert/function.py"
    click TestDeconv "https://github.com/compepigen/methylbert/blob/main/test/test_deconvolute.py"
    click TestFT "https://github.com/compepigen/methylbert/blob/main/test/test_finetune.py"
    click TestPre "https://github.com/compepigen/methylbert/blob/main/test/test_finetune_preprocess.py"
    click TestData "https://github.com/compepigen/methylbert/tree/main/test/data/"
    click T1 "https://github.com/compepigen/methylbert/blob/main/tutorials/01_Data_Preparation.md"
    click T2 "https://github.com/compepigen/methylbert/blob/main/tutorials/02_Preprocessing_training_data.ipynb"
    click T3 "https://github.com/compepigen/methylbert/blob/main/tutorials/03_Preprocessing_bulk_data.ipynb"
    click T4 "https://github.com/compepigen/methylbert/blob/main/tutorials/04_Fine-tuning_MethylBERT_model.ipynb"
    click T5 "https://github.com/compepigen/methylbert/blob/main/tutorials/05_tumour_deconvolution.ipynb"

    %% Styles
    classDef cli fill:#D291BC,stroke:#333,stroke-width:1px;
    classDef config fill:#B5A7D9,stroke:#333,stroke-width:1px;
    classDef data fill:#9EC1CF,stroke:#333,stroke-width:1px;
    classDef model fill:#F4A582,stroke:#333,stroke-width:1px;
    classDef inference fill:#B2D8B2,stroke:#333,stroke-width:1px;
    classDef util fill:#FFD1DC,stroke:#333,stroke-width:1px;
    classDef external fill:#F2E394,stroke:#333,stroke-width:1px;
    classDef io fill:#E8E9EB,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5;
    classDef test fill:#E6E6FA,stroke:#333,stroke-width:1px;
    classDef tutorial fill:#FFFACD,stroke:#333,stroke-width:1px;
```
