How kundajelab/chrombpnet works
Project type and purpose
• Type: An open-source, command-line–driven Python library and pipeline for training bias-factorized, base-resolution deep-learning models of chromatin accessibility.
• Purpose:
– Ingest raw chromatin assay data (BAM/fragment/tagAlign, genome FASTA & sizes, peaks/non-peaks).
– Preprocess reads into bigWig and motif files.
– Train a “bias model” to capture assay bias.
– Train the main ChromBPNet model (TF model) using bias correction.
– Interpret and evaluate models via notebooks/scripts.
– Generate reports (HTML/PDF) and visualization tracks.
Main components
A. User Interface & Orchestration
– CLI entrypoint (“chrombpnet” console script defined in setup.py)
– Bash workflows (workflows/*.sh, tests/*.sh)
– Dockerfile / Conda environment
B. Core Library (chrombpnet package)
1. Data ingestion & parsing
• chrombpnet/parsers.py, pipelines.py
2. Preprocessing helpers
• chrombpnet/helpers/preprocessing/…
• BigWig conversion, PWM generation, bias background regions
3. Model training
• chrombpnet/training/ (data_generators, models, train.py, predict.py, metrics.py)
4. Bias‐factorized logic
• chrombpnet/CHROMBPNET.py (orchestrates bias vs TF model)
5. Evaluation & interpretation
• chrombpnet/evaluation/ (figure_notebooks, interpret, modisco, footprints, variant_effect_prediction)
6. Reporting
• chrombpnet/helpers/generate_reports/
C. External Tools & Dependencies
– Bioinformatics binaries: samtools, bedtools, ucsc-bedgraphtobigwig, pybigwig, MEME suite
– Python frameworks: TensorFlow/Keras, numpy, pandas, h5py, shap
– Container/Environment: Docker, NVIDIA Container Toolkit, Conda
D. Data Stores & Artifacts
– Input: raw BAM/fragment/tagAlign, reference FASTA, chromosome sizes, peaks/non-peaks BED, JSON splits
– Models & logs: HDF5 models (bias.h5, chrombpnet.h5), log files
– Evaluation outputs: PDF/HTML reports, bigWig tracks, motif PDFs, footprints PNGs
Relationships and data flow
1. User invokes CLI or workflow script →
2. Data ingestion/parsers load raw inputs →
3. Preprocessing helpers convert reads → bigWigs, generate GC-matched negatives, motifs →
4. Bias pipeline (chrombpnet bias pipeline) trains bias.h5 →
5. ChromBPNet pipeline uses bias.h5 plus inputs to train TF model → outputs chrombpnet.h5 and chrombpnet_nobias.h5 →
6. Evaluation module reads models + test data → notebooks/scripts in evaluation/ produce figures, metrics →
7. Reporting helpers compile HTML/PDF in overall_report →
8. Final artifacts (models, logs, reports, tracks) are written to output directory.
Architectural patterns & design principles
• Modular package structure: clear separation between preprocessing, training, evaluation, reporting.
• Two-stage training (bias model first, then TF model) encapsulated in separate CLI pipelines.
• Use of data generators and modular Keras models (dilated convolutions, residual connections).
• Helper scripts and notebooks decoupled from core library for flexibility and extensibility.
• Pipeline orchestration via simple shell scripts rather than heavyweight workflow engines.
Technologies & frameworks to annotate
• Python 3.8+ environment (Conda or Docker)
• Deep learning: TensorFlow/Keras
• Bioinformatics: samtools, bedtools, UCSC utilities, pyBigWig, MEME
• Container: Docker + NVIDIA Container Toolkit
• Reporting: matplotlib/seaborn in notebooks, HTML via make_html.py