How sanofi-public/mrna-lm works
Project type and purpose
• This is a machine-learning fine-tuning pipeline for mRNA-based language models (CodonBERT, 5UTRBERT, 3UTRBERT).
• Purpose: load pre-trained transformer models, fine-tune on specific tasks (e.g., mRNA half-life prediction), and output trained checkpoints.
File structure
• Top-level scripts:
• dataload.py – loads and preprocesses CSV datasets
• FullModel.py & OneModel.py – define neural architectures (multi-task vs single-task heads)
• finetune_all.py – fine-tune one model on all tasks or multiple heads
• finetune_each.py – fine-tune separate models per task
• data/ – raw CSVs for tasks
• requirements.txt – dependencies
• README.md – usage and CLI args
README for additional insights
• CLI-based training entry points
• Pre-trained model download URLs
• Fine-tune parameters (batch size, LR, LoRA hyperparameters, CLIP loss options)
a. Main components:
1. External storage (pre-trained model artifacts – CodonBERT, 5UTRBERT, 3UTRBERT)
2. Data source (CSV files under data/)
3. DataLoader module (dataload.py)
4. Model definitions (FullModel.py and OneModel.py)
5. Training orchestrators (finetune_all.py, finetune_each.py)
6. GPU compute resource
7. Output directory (trained checkpoints, logs)
b. Relationships and interactions:
• CLI scripts invoke DataLoader → preprocess and batch data
• DataLoader feeds batches to Model
• Model loads pre-trained weights from external storage
• Training scripts coordinate forward/backward passes on GPU
• Checkpoints & metrics written to output folder
c. Architectural patterns and design principles:
• Modular separation: Data ingestion, Model definition, Training loop
• Support for multi-task vs single-task heads via separate model classes
• Parameterization via CLI arguments for hyperparameters (loRA, CLIP, LR, etc.)
d. Key technologies:
• PyTorch (implicit from model & training scripts)
• LoRA (low-rank adaptation) for fine-tuning
• CLIP loss integration (optional)
• CSV parsing (pandas or built-in)
Detail without over-compartmentalizing
• Six components above
• CLI entry points and key flags (–task, –lr, –lorar, –useCLIP)