# kundajelab/chrombpnet architecture

> Architecture diagram of the GitHub repository kundajelab/chrombpnet, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kundajelab/chrombpnet
- Repository: https://github.com/kundajelab/chrombpnet

## Overview

##### Project type and purpose
- Type: An open-source, command-line–driven Python library and pipeline for training bias-factorized, base-resolution deep-learning models of chromatin accessibility.
- Purpose:
– Ingest raw chromatin assay data (BAM/fragment/tagAlign, genome FASTA & sizes, peaks/non-peaks).
– Preprocess reads into bigWig and motif files.
– Train a “bias model” to capture assay bias.
– Train the main ChromBPNet model (TF model) using bias correction.
– Interpret and evaluate models via notebooks/scripts.
– Generate reports (HTML/PDF) and visualization tracks.

##### Main components
A. User Interface & Orchestration
– CLI entrypoint (“chrombpnet” console script defined in setup.py)
– Bash workflows (workflows/*.sh, tests/*.sh)
– Dockerfile / Conda environment
B. Core Library (chrombpnet package)
1. Data ingestion & parsing
- chrombpnet/parsers.py, pipelines.py
2. Preprocessing helpers
- chrombpnet/helpers/preprocessing/…
- BigWig conversion, PWM generation, bias background regions
3. Model training
- chrombpnet/training/ (data_generators, models, train.py, predict.py, metrics.py)
4. Bias‐factorized logic
- chrombpnet/CHROMBPNET.py (orchestrates bias vs TF model)
5. Evaluation & interpretation
- chrombpnet/evaluation/ (figure_notebooks, interpret, modisco, footprints, variant_effect_prediction)
6. Reporting
- chrombpnet/helpers/generate_reports/
C. External Tools & Dependencies
– Bioinformatics binaries: samtools, bedtools, ucsc-bedgraphtobigwig, pybigwig, MEME suite
– Python frameworks: TensorFlow/Keras, numpy, pandas, h5py, shap
– Container/Environment: Docker, NVIDIA Container Toolkit, Conda
D. Data Stores & Artifacts
– Input: raw BAM/fragment/tagAlign, reference FASTA, chromosome sizes, peaks/non-peaks BED, JSON splits
– Models & logs: HDF5 models (bias.h5, chrombpnet.h5), log files
– Evaluation outputs: PDF/HTML reports, bigWig tracks, motif PDFs, footprints PNGs

##### Relationships and data flow
1. User invokes CLI or workflow script →
2. Data ingestion/parsers load raw inputs →
3. Preprocessing helpers convert reads → bigWigs, generate GC-matched negatives, motifs →
4. Bias pipeline (chrombpnet bias pipeline) trains bias.h5 →
5. ChromBPNet pipeline uses bias.h5 plus inputs to train TF model → outputs chrombpnet.h5 and chrombpnet_nobias.h5 →
6. Evaluation module reads models + test data → notebooks/scripts in evaluation/ produce figures, metrics →
7. Reporting helpers compile HTML/PDF in overall_report →
8. Final artifacts (models, logs, reports, tracks) are written to output directory.

##### Architectural patterns & design principles
- Modular package structure: clear separation between preprocessing, training, evaluation, reporting.
- Two-stage training (bias model first, then TF model) encapsulated in separate CLI pipelines.
- Use of data generators and modular Keras models (dilated convolutions, residual connections).
- Helper scripts and notebooks decoupled from core library for flexibility and extensibility.
- Pipeline orchestration via simple shell scripts rather than heavyweight workflow engines.

##### Technologies & frameworks to annotate
- Python 3.8+ environment (Conda or Docker)
- Deep learning: TensorFlow/Keras
- Bioinformatics: samtools, bedtools, UCSC utilities, pyBigWig, MEME
- Container: Docker + NVIDIA Container Toolkit
- Reporting: matplotlib/seaborn in notebooks, HTML via make_html.py

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Interface & Orchestration Layer
    subgraph "User Interface & Orchestration"
        direction TB
        CLI["chrombpnet CLI"]:::ui
        Workflows["Bash Workflows"]:::ui
        DockerEnv["Docker / Conda Environment"]:::ui
    end

    %% Core Library
    subgraph "Core Library (chrombpnet)"
        direction TB
        subgraph "Data Preparation"
            direction TB
            Ingestion["Data Ingestion & Parsing"]:::core
            Preproc["Preprocessing Helpers"]:::core
            RefData["Reference Data & Motifs"]:::core
            SplitsGC["Chr Splits & GC‐Matched Negatives"]:::core
            Hyperparam["Hyperparameter Selection"]:::core
            Postproc["Postprocessing Outputs"]:::core
        end

        subgraph "Model Training"
            direction TB
            BiasOrch["Bias‐Model Orchestration"]:::core
            subgraph "Bias & TF Training Components"
                direction TB
                DataGen["Data Generators"]:::core
                Models["Model Definitions"]:::core
                TrainPred["Training & Prediction Scripts"]:::core
                Metrics["Metrics & Utilities"]:::core
            end
        end

        subgraph "Evaluation & Reporting"
            direction TB
            Eval["Evaluation & Interpretation"]:::core
            Reporting["Reporting Helpers"]:::core
        end
    end

    %% External Tools & Dependencies
    subgraph "External Tools & Dependencies"
        direction TB
        BioTools["samtools, bedtools, UCSC utils, MEME"]:::ext
        PyLibs["TensorFlow/Keras, numpy, pandas, shap"]:::ext
        Container["Docker + NVIDIA, Conda"]:::ext
    end

    %% Data Stores & Artifacts
    subgraph "Data Stores & Artifacts"
        direction TB
        RawData["Raw Inputs (.bam, .fasta, .bed, JSON)"]:::ds
        Intermediate["Intermediate Files (.bigwig, motifs)"]:::ds
        ModelsOut["Models & Logs (.h5, logs)"]:::ds
        Reports["Reports & Tracks (HTML/PDF/png)"]:::ds
    end

    %% Connections
    CLI -->|invokes| Ingestion
    Workflows -->|runs scripts| Ingestion

    Ingestion -->|".bam → .bigwig"| Preproc
    Preproc -->|bigWig, PWM| Intermediate

    Intermediate -->|input| BiasOrch
    BiasOrch -->|bias.h5| ModelsOut

    ModelsOut -->|use bias.h5| TrainPred
    TrainPred -->|chrombpnet.h5| ModelsOut

    ModelsOut -->|test data| Eval
    Eval -->|figures, metrics| Reports
    Reporting -->|compile| Reports

    %% Dependencies
    Ingestion --> BioTools
    Preproc --> BioTools
    BiasOrch --> PyLibs
    TrainPred --> PyLibs
    Reporting --> PyLibs
    CLI --> Container

    %% Data Stores
    RawData --> Ingestion
    ModelsOut --> Eval
    Reports --> Reports

    %% Click Events
    click CLI "https://github.com/kundajelab/chrombpnet/blob/master/setup.py"
    click Workflows "https://github.com/kundajelab/chrombpnet/blob/master/workflows/train_bias_model.sh"
    click Workflows "https://github.com/kundajelab/chrombpnet/blob/master/workflows/train_chrombpnet_model.sh"
    click Workflows "https://github.com/kundajelab/chrombpnet/tree/master/workflows/tutorial/"
    click Workflows "https://github.com/kundajelab/chrombpnet/tree/master/tests/"
    click DockerEnv "https://github.com/kundajelab/chrombpnet/tree/master/Dockerfile"
    click DockerEnv "https://github.com/kundajelab/chrombpnet/blob/master/requirements.txt"
    click Ingestion "https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/parsers.py"
    click Ingestion "https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/pipelines.py"
    click Preproc "https://github.com/kundajelab/chrombpnet/tree/master/chrombpnet/helpers/preprocessing/"
    click RefData "https://github.com/kundajelab/chrombpnet/tree/master/chrombpnet/data/"
    click SplitsGC "https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/helpers/make_chr_splits/splits.py"
    click SplitsGC "https://github.com/kundajelab/chrombpnet/tree/master/chrombpnet/helpers/make_gc_matched_negatives/"
    click Hyperparam "https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/helpers/hyperparameters/find_bias_hyperparams.py"
    click Hyperparam "https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/helpers/hyperparameters/find_chrombpnet_hyperparams.py"
    click Hyperparam "https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/helpers/hyperparameters/param_utils.py"
    click Postproc "https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/helpers/postprocessing/reformat_chrombpnet_h5.py"
    click BiasOrch "https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/CHROMBPNET.py"
    click DataGen "https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/training/data_generators/batchgen_generator.py"
    click DataGen "https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/training/data_generators/initializers.py"
    click Models "https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/training/models/bpnet_model.py"
    click Models "https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/training/models/chrombpnet_with_bias_model.py"
    click TrainPred "https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/training/train.py"
    click TrainPred "https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/training/predict.py"
    click Metrics "https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/training/metrics.py"
    click Metrics "https://github.com/kundajelab/chrombpnet/tree/master/chrombpnet/training/utils/"
    click Eval "https://github.com/kundajelab/chrombpnet/tree/master/chrombpnet/evaluation/"
    click Reporting "https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/helpers/generate_reports/make_html.py"
    click Reporting "https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/helpers/generate_reports/make_html_bias.py"

    %% Styles
    classDef ui fill:#e0f7e9,stroke:#2ecc71,color:#27ae60;
    classDef core fill:#e8f4fd,stroke:#3498db,color:#2980b9;
    classDef ext fill:#fff0e0,stroke:#e67e22,color:#d35400;
    classDef ds fill:#f2f2f2,stroke:#7f8c8d,color:#95a5a6;
```
