How jkobject/geneformer works
Project Type and Purpose
• Geneformer is a deep learning project that deals with single cell transcriptomes. It provides a foundation transformer model pretrained on a massive corpus (~30M cells) to support downstream tasks like fine-tuning for cell state classification, disease classification, in silico perturbation, etc.
• The project includes both training and inference components as well as standalone tools for tokenizing, collating data, pretraining, and fine-tuning.
Analyzing the File Structure
• Key top-level entries:
• The “geneformer” directory holds the core library code (e.g., tokenizer, pretrainer, in silico perturber) and associated resources (e.g., pickle files for gene dictionaries).
• The “examples” directory contains Jupyter notebooks and scripts that demonstrate different usage scenarios such as cell classification, gene classification, hyperparameter optimization, and in silico perturbation.
• The “fine_tuned_models” directory contains pretrained and fine-tuned checkpoints (model weights, optimizer and trainer state, configuration files) used for downstream tasks.
• Files like “setup.py” and “config.json” indicate package setup and configuration.
• The structure is modular: a core library, example usage scripts, and checkpoints/results. This implies clear separation between the model’s implementation (core transformer logic and utility functions) and usage (examples/fine-tuned models).
Insights from the README
• The README clearly outlines the scientific background, model design (six transformer encoder layers for the base model and a 12-layer variant), as well as the pretraining strategy (self-supervised masked learning based on rank encoding of gene expression data).
• It points out key application areas, emphasizing both zero-shot learning and fine-tuning scenarios.
• The integration with external datasets (Genecorpus-30M from Hugging Face) and the reliance on GPU resources are also highlighted.
Main Components to Capture
a. Data Ingestion and Preprocessing:
• Data Source: Genecorpus-30M (external dataset)
• Preprocessing: Tokenization and rank-value encoding (handled by tokenizing modules such as geneformer/tokenizer.py and utils related to collating)
b. Core Model Infrastructure:
• Core Library (located in the “geneformer” directory):
• Pretrainer module (geneformer/pretrainer.py) for masked learning.
• Transformer architecture details (6-layer and 12-layer implementations).
• Supporting modules for extracting embeddings and handling in silico perturbations.
c. Training and Fine-tuning Pipeline:
• Pretraining process: Converts input transcriptomes, applies masked learning, and outputs pre-trained model checkpoints.
• Fine-tuning components: Additional scripts and fine-tuned model checkpoints (in fine_tuned_models) adjusted for task-specific applications.
d. Downstream Application Modules:
• In Silico Perturbation: Modules for simulation (geneformer/in_silico_perturber.py and related scripts).
• Example Notebooks: Demonstrations of tasks (classification, clustering, hyperparameter tuning) that show how to deploy the pretrained/fine-tuned models.
e. Model Deployment/Usage:
• Inference usage for zero-shot analysis and downstream prediction tasks.
• Integration points such as using the pre-trained models from Hugging Face via git-lfs.
Relationships and Data Flow
• Separation between the training pipeline and downstream application modules, where the output of the pretraining and fine-tuning modules serves as input to tasks like cell classification and in silico analysis.
• Iterative fine-tuning process where a pretrained model is adjusted based on task-specific data.
Architectural Patterns and Important Technologies
• Modular design: Separation into distinct modules for preprocessing, core model training, and downstream applications.
• Self-supervised learning: The pretraining strategy is a key design principle.
• Use of deep learning frameworks (PyTorch is indicated by files such as pytorch_model.bin, training_args.bin, optimizer.pt, scheduler.pt) and tools like git-lfs for large model files.
• Notebooks and scripts in “examples” indicate an experimental or research-based workflow.