How frogstar-world-b/transfer-learning-chemberta works
Main Components
• Data Sources: Represent the “data” folder containing Excel and CSV files. These files are the raw input containing chemical information and pIC50 values.
• Preprocessing & Exploratory Data Analysis (EDA): Capture the functionality implemented in preprocessing.ipynb. This component is responsible for cleaning the data, extracting SMILES information, and preparing the dataset.
• Modeling Approaches:
a. Fine-Tuning Module: Corresponds to fine-tune.ipynb, where the ChemBERTa model is further trained using the Transformers library.
b. Transfer Learning Module: Reflected in transfer-learning.ipynb that uses pre-trained ChemBERTa embeddings along with an XGBoost regressor.
c. Extended Transfer Learning with Descriptors: Represented by transfer-learning-plus-descriptors.ipynb, which further enhances the transfer learning pipeline by incorporating molecular descriptors.
• Evaluation and Metrics: A logical part of the process where performance metrics like Mean Squared Error (MSE) and Spearman Correlation are computed to compare the approaches.
Denote Relationships and Interactions
• Model Pipeline: From preprocessed data, branch out into both the fine-tuning module and the transfer learning module (with an extra branch to incorporate molecular descriptors).
• Feedback/Results Loop: Connect model outputs to an evaluation component which then may feed insights back into an iteration of preprocessing or hyperparameter tuning.
Architectural Patterns and Design Principles
• Modularity: Even though the project is packaged as Jupyter notebooks, the design demonstrates separation of concerns by isolating data preprocessing, model training, and evaluation tasks.
• Experimentation vs. Production: The structure is set up for comparative experimentation. Capture the idea of parallel pipelines (fine-tuning vs. transfer learning) and highlight points where these pipelines differ and converge.
• Integration of Heterogeneous Technologies: Emphasize that the architecture uses multiple libraries and frameworks (PyTorch, Transformers, XGBoost, RDKit), each fulfilling specific roles (deep learning, regression, chemical informatics).
Relevant Technologies and Frameworks
• PyTorch and Hugging Face Transformers: Used within the fine-tuning workflow.
• XGBoost: Employed in the transfer-learning pipelines.
• RDKit: Critical for handling and processing chemical information.
• Standard Python Data Science Stack: NumPy, pandas, scikit-learn, Matplotlib, etc., which operate under the hood for EDA and evaluation.
Elements
• Clear Labels: Each component (Data Ingestion, Preprocessing & EDA, Fine-Tuning, Transfer Learning, Extended Transfer Learning, Evaluation) should be explicitly labeled.
• Begin on the left with “Data Sources” (show data/hDHFR.xlsx and data/hDHFR_pIC50_data.csv).
• Split the workflow into two parallel paths:
• Fine-Tuning Path: Illustrate the fine-tuning process using ChemBERTa with a block labeled “Fine-Tuning ChemBERTa” (from fine-tune.ipynb), leveraging PyTorch & Transformers.
• Transfer Learning Path:
• One branch for “Transfer Learning – ChemBERTa Embeddings” (from transfer-learning.ipynb) using XGBoost.
• Another branch for “Transfer Learning with Descriptors” (from transfer-learning-plus-descriptors.ipynb) which integrates additional molecular descriptors.
• Converge these pipelines into an “Evaluation” component where metrics (MSE, Spearman Correlation) are computed and compared.
• Optionally, include a side note that hyperparameter tuning is an iterative process which might loop back to the model training blocks.