# jkobject/geneformer architecture

> Architecture diagram of the GitHub repository jkobject/geneformer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jkobject/geneformer
- Repository: https://github.com/jkobject/geneformer

## Overview

##### Project Type and Purpose
- Geneformer is a deep learning project that deals with single cell transcriptomes. It provides a foundation transformer model pretrained on a massive corpus (~30M cells) to support downstream tasks like fine-tuning for cell state classification, disease classification, in silico perturbation, etc.
- The project includes both training and inference components as well as standalone tools for tokenizing, collating data, pretraining, and fine-tuning.

##### Analyzing the File Structure
- Key top-level entries:
- The “geneformer” directory holds the core library code (e.g., tokenizer, pretrainer, in silico perturber) and associated resources (e.g., pickle files for gene dictionaries).
- The “examples” directory contains Jupyter notebooks and scripts that demonstrate different usage scenarios such as cell classification, gene classification, hyperparameter optimization, and in silico perturbation.
- The “fine_tuned_models” directory contains pretrained and fine-tuned checkpoints (model weights, optimizer and trainer state, configuration files) used for downstream tasks.
- Files like “setup.py” and “config.json” indicate package setup and configuration.
- The structure is modular: a core library, example usage scripts, and checkpoints/results. This implies clear separation between the model’s implementation (core transformer logic and utility functions) and usage (examples/fine-tuned models).

##### Insights from the README
- The README clearly outlines the scientific background, model design (six transformer encoder layers for the base model and a 12-layer variant), as well as the pretraining strategy (self-supervised masked learning based on rank encoding of gene expression data).
- It points out key application areas, emphasizing both zero-shot learning and fine-tuning scenarios.
- The integration with external datasets (Genecorpus-30M from Hugging Face) and the reliance on GPU resources are also highlighted.

##### Main Components to Capture
a. Data Ingestion and Preprocessing:
- Data Source: Genecorpus-30M (external dataset)
- Preprocessing: Tokenization and rank-value encoding (handled by tokenizing modules such as geneformer/tokenizer.py and utils related to collating)
b. Core Model Infrastructure:
- Core Library (located in the “geneformer” directory):
- Pretrainer module (geneformer/pretrainer.py) for masked learning.
- Transformer architecture details (6-layer and 12-layer implementations).
- Supporting modules for extracting embeddings and handling in silico perturbations.
c. Training and Fine-tuning Pipeline:
- Pretraining process: Converts input transcriptomes, applies masked learning, and outputs pre-trained model checkpoints.
- Fine-tuning components: Additional scripts and fine-tuned model checkpoints (in fine_tuned_models) adjusted for task-specific applications.
d. Downstream Application Modules:
- In Silico Perturbation: Modules for simulation (geneformer/in_silico_perturber.py and related scripts).
- Example Notebooks: Demonstrations of tasks (classification, clustering, hyperparameter tuning) that show how to deploy the pretrained/fine-tuned models.
e. Model Deployment/Usage:
- Inference usage for zero-shot analysis and downstream prediction tasks.
- Integration points such as using the pre-trained models from Hugging Face via git-lfs.

##### Relationships and Data Flow
- Separation between the training pipeline and downstream application modules, where the output of the pretraining and fine-tuning modules serves as input to tasks like cell classification and in silico analysis.
- Iterative fine-tuning process where a pretrained model is adjusted based on task-specific data.

##### Architectural Patterns and Important Technologies
- Modular design: Separation into distinct modules for preprocessing, core model training, and downstream applications.
- Self-supervised learning: The pretraining strategy is a key design principle.
- Use of deep learning frameworks (PyTorch is indicated by files such as pytorch_model.bin, training_args.bin, optimizer.pt, scheduler.pt) and tools like git-lfs for large model files.
- Notebooks and scripts in “examples” indicate an experimental or research-based workflow.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        DS["Genecorpus-30M (External Dataset)"]:::external
    end

    %% Preprocessing Layer
    subgraph "Preprocessing Layer"
        TKN["Tokenizer Module"]:::utility
        DC["Data Collator"]:::utility
        TNB["Tokenizing Notebook"]:::utility
    end

    %% Core Model Layer
    subgraph "Core Model Layer"
        PT["Pretrainer Module"]:::core
        ISP["In Silico Perturber"]:::core
        ISS["In Silico Perturber Stats"]:::core
        EE["Embedding Extraction"]:::core
    end

    %% Training & Fine-tuning Layer
    subgraph "Training & Fine-tuning"
        PS["Pretraining Script"]:::training
        PN["Pretraining Notebook"]:::training
        FTC["Fine-tuning Checkpoints"]:::training
    end

    %% Downstream Applications
    subgraph "Downstream Applications"
        CC["Cell Classification Notebook"]:::application
        GCN["Gene Classification Notebook"]:::application
        HO["Hyperparameter Optimization Script"]:::application
        IPN["In Silico Perturbation Notebook"]:::application
    end

    %% Model Deployment/Usage
    subgraph "Model Deployment/Usage"
        PMC["Pretrained Model Config"]:::deployment
        PTA["Pretrained Model Training Args"]:::deployment
        CI["Core Package Init"]:::deployment
        SS["Setup Script"]:::deployment
    end

    %% Connections from Data Layer to Preprocessing Layer
    DS --> TKN
    DS --> DC
    DS --> TNB

    %% Connections from Preprocessing Layer to Core Model Layer
    TKN --> PT
    DC --> PT
    TNB --> PT

    %% Connections within Core Model Layer
    PT --> ISP
    PT --> ISS
    PT --> EE

    %% Connections from Core Model Layer to Training & Fine-tuning Layer
    ISP --> PS
    ISS --> PS
    EE --> PS
    PT --> PN

    %% Connections within Training & Fine-tuning Layer
    PS --> FTC
    PN --> FTC

    %% Connections from Training & Fine-tuning to Downstream Applications
    FTC --> CC
    FTC --> GCN
    FTC --> HO
    FTC --> IPN

    %% Connections from Core Model Layer and Training Layer to Deployment Layer
    PT --> PMC
    PT --> PTA
    FTC --> PMC
    FTC --> PTA

    %% Connections from Downstream Applications to Deployment Layer
    CC --> CI
    GCN --> CI
    HO --> CI
    IPN --> CI
    CI --> SS

    %% Click Events for Preprocessing Layer
    click TKN "https://github.com/jkobject/geneformer/blob/main/geneformer/tokenizer.py"
    click DC "https://github.com/jkobject/geneformer/blob/main/geneformer/collator_for_classification.py"
    click TNB "https://github.com/jkobject/geneformer/blob/main/examples/tokenizing_scRNAseq_data.ipynb"
    
    %% Click Events for Core Model Layer
    click PT "https://github.com/jkobject/geneformer/blob/main/geneformer/pretrainer.py"
    click ISP "https://github.com/jkobject/geneformer/blob/main/geneformer/in_silico_perturber.py"
    click ISS "https://github.com/jkobject/geneformer/blob/main/geneformer/in_sico_perturber_stats.py"
    click EE "https://github.com/jkobject/geneformer/blob/main/geneformer/emb_extractor.py"
    
    %% Click Events for Training & Fine-tuning Layer
    click PS "https://github.com/jkobject/geneformer/blob/main/examples/pretraining_new_model/pretrain_geneformer_w_deepspeed.py"
    click PN "https://github.com/jkobject/geneformer/blob/main/examples/pretraining_new_model/obtain_nonzero_median_digests.ipynb"
    click FTC "https://github.com/jkobject/geneformer/tree/main/fine_tuned_models/geneformer-6L-30M_CellClassifier_cardiomyopathies_220224"
    
    %% Click Events for Downstream Applications
    click CC "https://github.com/jkobject/geneformer/blob/main/examples/cell_classification.ipynb"
    click GCN "https://github.com/jkobject/geneformer/blob/main/examples/gene_classification.ipynb"
    click HO "https://github.com/jkobject/geneformer/blob/main/examples/hyperparam_optimiz_for_disease_classifier.py"
    click IPN "https://github.com/jkobject/geneformer/blob/main/examples/in_silico_perturbation.ipynb"
    
    %% Click Events for Model Deployment/Usage
    click PMC "https://github.com/jkobject/geneformer/blob/main/geneformer-12L-30M/config.json"
    click PTA "https://github.com/jkobject/geneformer/blob/main/geneformer-12L-30M/training_args.bin"
    click CI "https://github.com/jkobject/geneformer/blob/main/geneformer/__init__.py"
    click SS "https://github.com/jkobject/geneformer/blob/main/setup.py"

    %% Class Definitions
    classDef external fill:#ffe0f0,stroke:#cc0099,stroke-width:2px;
    classDef utility fill:#e0f7ff,stroke:#0077cc,stroke-width:2px;
    classDef core fill:#e0ffe0,stroke:#009933,stroke-width:2px;
    classDef training fill:#fff0b3,stroke:#ff9933,stroke-width:2px;
    classDef application fill:#ffd9ff,stroke:#cc66ff,stroke-width:2px;
    classDef deployment fill:#d9ffff,stroke:#33cccc,stroke-width:2px;
```
