How facebookresearch/esm works
Below are the steps and components to include:
Overall Project Type and Purpose
• The README tells you that this repository is a research and production codebase for state‐of‐the‐art transformer models in computational biology. It includes implementations for multiple models (e.g., ESM‑1, ESM‑2, ESMFold, ESM‑1v, and ESM‑IF1), targeting tasks such as structure prediction, variant effect prediction, inverse folding, and even generative protein design.
• Given the file tree structure with core libraries (esm/), examples (demonstrating different use cases), scripts (for command-line tools like embedding extraction and folding operations), and tests, the project is essentially an open‑source toolkit for protein modeling based on transformer neural networks.
And Segment the File Structure into Main Components
• Core Library (“esm” package):
– Contains submodules and key components:
▸ esm/__init__.py – main initialization.
▸ esm/axial_attention.py, esm/multihead_attention.py, esm/rotary_embedding.py, esm/constants.py, esm/data.py – provide low-level utilities and functions related to transformer operations.
▸ esm/model/ – includes model definitions (esm1.py, esm2.py, msa_transformer.py) indicating support for multiple transformer variants.
▸ esm/esmfold/ – houses the ESMFold module (v1 directory contains files like esmfold.py, pretrained.py, etc.). This branch integrates with structure prediction and has dependencies on OpenFold.
▸ esm/inverse_folding/ – contains code related to the inverse folding model (files such as transformer_decoder.py, gvp_encoder.py, etc.).
– Additional utility modules are found in files like esm/modules.py and esm/version.py.
• Pretrained Model Interface:
– Pretrained models are exposed through a high-level API (via functions in esm.pretrained) and also integrated with PyTorch Hub.
• Example Applications:
– The examples directory is subdivided into several folders (e.g., lm-design, protein-programming-language, inverse_folding, variant-prediction, contact_prediction, sup_variant_prediction, etc.). These notebooks and scripts show how to use the core library for diverse tasks like structure prediction, sequence design, contact mapping, and variant effect prediction.
• Scripts and Command-Line Interfaces:
– The scripts directory includes utilities for bulk inference, embedding extraction, atlas building, downloading weights, and even non-code operations (like fold predictions). The “atlas” subdirectory, for instance, points to bulk downloads and usage of metagenomic predictions.
• Configuration and Testing:
– Files such as setup.py, pyproject.toml, environment.yml, and .flake8 coordinate installation, dependency management, and coding style.
– The tests/ directory contains unit tests that validate critical functionality.
Relationships and Interactions Between Components
• The Core Library (esm) provides low-level building blocks (e.g. attention modules, data pipelines, transformer layers) that are used by higher-level modules such as ESMFold and the inverse folding models.
• The “pretrained” interface is an abstraction layer that downloads and loads large-scale pre-trained weights and configurations.
• The Examples and Scripts are “client” applications that call into the esm core APIs; they illustrate usage patterns and workflows (e.g., how to perform structure prediction with esmfold_v1 or how to score protein variants using ESM‑1v).
• Testing modules validate both individual components and integration of larger functionalities.
Architectural Patterns & Design Principles
• Modularity: The project is split into clearly separated packages (esm core, examples, scripts) that decouple model definitions from usage examples.
• Layered Architecture: There is a lower layer (utility functions, attention modules, etc.), an intermediate layer (model architectures including language models and structure modules), and an upper layer (application scripts, notebooks, and pre-trained model interfaces).
• Extensibility: The design supports multiple transformer model variants (ESM‑1, ESM‑2, MSA Transformer, etc.) and protocols (structure prediction, inverse folding, variant prediction), which indicates strong modularity and plug‑in possibilities.
• Reusability: Common utilities (data handling, tokenization, batch conversion) are centralized in the esm package, enabling reuse across models and examples.
Relevant Technologies, External Services, & Frameworks
• The project is built on Python and uses PyTorch for deep learning.
• It integrates with external libraries such as OpenFold (for leveraging ESMFold), HuggingFace Transformers (for streamlined model usage), and CUDA for GPU acceleration.
• There is support for FSDP (Fully Sharded Data Parallel via FairScale) for scalable model inference.
• Configuration and environment management (environment.yml, setup.py) indicate that conda and pip are used.
Be Detailed but Organized
• To include external dependency flows (e.g., callouts to OpenFold for ESMFold, or integration with cloud APIs for the Metagenomic Atlas).