How bytedance/dplm works
Project Type & Purpose
• This repository implements DPLM as described in the ICML’24 paper “Diffusion Language Models Are Versatile Protein Learners.”
• It is a full-scale deep learning project working with protein sequences – including functionalities for unconditional generation, motif scaffolding, inverse folding, and representation learning.
• Key aspects include model training (using PyTorch Lightning, Hydra for configuration, and distributed training), data processing (with custom datamodules for protein datasets like UniRef50 and CATH), and diverse evaluation pipelines (using analysis scripts and notebooks).
File Structure Analysis & Main Architectural Areas
• “configs/”: Contains various YAML configuration files – for callbacks, data modules, experiments, hydra configuration, and trainer setups. This indicates a heavily configurable architecture.
• “src/byprot/”: Holds the core implementation in a modular format. Within this folder:
• “datamodules/” handles data loading and preprocessing for different datasets.
• “models/lm/” defines model variants (dplm, cond_dplm, esm_dplm) and modules (adapter modules, transformer encoders).
• “tasks/lm/” contains scripts defining different training/inference tasks such as conditional and unconditional generation.
• “utils/” provides configurations, I/O, logging, learning rate scheduling, optimization, and strategy registries.
• “modules/” includes common components like cross entropy, metrics, and protein-specific metrics.
• “analysis/”: Holds scripts and notebooks for downstream analysis and evaluation (e.g., for generating plots, calculating scores).
• “scripts/” and “run/”: Provide auxiliary deployment and running scripts for downloading datasets and executing generation pipelines.
• “train.py” and “test.py”: Script entry points for training and testing pipelines.
• “generate.py” and “scaffold_generate.py”: Dedicated to protein sequence generation tasks.
a. Main Components:
• Data Ingestion and Preprocessing:
• Data modules for UniRef50 and CATH (located in “configs/datamodule” and “src/byprot/datamodules”).
• Download scripts and auxiliary preprocessed data (in “scripts/” & “data-bin/”).
• Model Definition & Modules:
• Core diffusion language model (DPLM) implementations (in “src/byprot/models/lm”).
• Submodules such as adapter modules and transformer encoders (in “src/byprot/models/lm/modules”).
• Training & Task Pipelines:
• Training pipeline script (“train.py”) and different task definitions (conditional/unconditional generation, inverse folding) in “src/byprot/tasks/lm/”.
• Utility functions (in “src/byprot/utils”).
• Configuration Management:
• Extensive configurations managed via Hydra (files in “configs/hydra”, “configs/experiment”, “configs/trainer”, etc.).
• Evaluation & Analysis:
• Analysis scripts for metrics calculation (e.g., pLDDT evaluation, motif RMSD) in “analysis/”.
• Notebooks for visualization (motif_analysis.ipynb, uncond_analysis.ipynb).
• Generation Modules:
• Scripts for generating protein sequences (“generate.py”, “scaffold_generate.py”, “run/” directory scripts).
b. Relationships and Interactions:
• The configuration management layer interacts with every part (data modules, model structure, training strategy, logging).
• Model definitions in “src/byprot/models/” are used by both the training tasks and by generation scripts.
• Utility modules provide common functionality (I/O, logging, strategies) and are accessed by multiple components.
• Analysis modules take generation outputs and trained model checkpoints as inputs for evaluation metrics and downstream assessments.
• External dependencies like PyTorch Lightning, Hydra, and ESMFold (for structure prediction in analysis) should be noted as integrations.
c. Architectural Patterns & Design Principles:
• Modular design: Clear separation of concerns (data, model, training, evaluation).
• Configuration-driven development: Hydra configuration files allow customization across experiments.
• Reusability: Shared utilities and module components are designed for multiple tasks (generation, representation learning).
• Distributed training: Configurations and scripts supporting multi-GPU/multi-node training.
d. Relevant Technologies & Frameworks:
• PyTorch and PyTorch Lightning for training and model implementation.
• Hydra for configuration management.
• ESMFold for secondary evaluation of generated protein structures.
• External tools for data downloading and processing (scripts in “scripts/”).