How biomap-research/scfoundation works
Project Type and Purpose
• This is a research‐oriented, multi‐task machine‐learning repository centered on a large‐scale pretrained single‐cell transcriptomics foundation model (“scFoundation”).
• It provides pipelines for (a) pre‐training, (b) data preprocessing, (c) embedding inference, and (d) downstream tasks (DeepCDR, SCAD, GEARS, read‐depth enhancement, annotation, mapping, gene‐module inference, ablation studies, API examples).
• It is not a classical full‐stack web app; rather it is a modular ML toolkit with data, model code, scripts, notebooks, published weights, and CLI wrappers.
Key Components (to include
A. Data Sources & Stores
• Raw single‐cell data repository (preprocessing/data/original & Figshare links)
• Processed data (preprocessing/data/processing, SCAD/data/split_norm, DeepCDR/data, GEARS/data/demo)
• Published pretrained weights (model/pretrainmodels & DeepCDR/published)
• Task‐specific result folders (SCAD/results, enhancement outputs, annotation data)
B. Preprocessing Pipeline
• Entry point: preprocessing/scRNA_workflow.py & demo.sh
• Downloads & normalizes raw data → produces normalized gene expression matrices
C. Pre‐training & Embedding Generation
• Core model code (model/ folder) implementing xTrimoGene variants (mae_autobin, performer, transformer, reversible)
• Training script(s) (model/finetune_model.py, train with DeepSpeed/PyTorch Lightning)
• Embedding extraction (model/get_embedding.py, model/load.py)
• Output: multi‐million‐cell embeddings (50M‐0.1B‐res_embedding.npy)
D. Inference & API
• CLI example (apiexample/client.py, example.sh) calling new inference endpoint (https://aigp.biomap.com/)
• Local inference utilities in model/ or GEARS/inference.py
E. Downstream Task Modules
• DeepCDR folder: processes bulk data + embeddings → drug response prediction
• SCAD folder: single‐cell drug sensitivity prediction using embedding inputs
• GEARS folder: perturbation prediction; generates gene embeddings in training
• enhancement folder: read‐depth enhancement via embeddings
• annotation folder: cell‐type annotation pipelines
• mapping folder: organoid → in vivo mapping demos
• genemodule folder: infers gene modules from gene context embeddings
• ablation folder: comparative studies of model & loss variants
F. Utilities & Scripts
• Shell scripts (run.sh in each task folder) for reproducible experiments
• Jupyter notebooks for data exploration & plotting
Relationships & Data Flow
• Raw data → Preprocessing → Processed data
• Processed data + pretraining code → scFoundation weights + cell/gene embeddings
• Embeddings (and related processed data) feed into each downstream module
• Downstream modules produce results/figures and write to their results folders
• API client interacts with a remote inference service using the same embedding models
• Notebooks consume outputs at each stage for visualization
Architectural Patterns & Principles
• Modularization by task: each downstream experiment lives in its own folder with clear inputs, code, and outputs
• Reuse of core model code for both pretraining and inference across tasks
• Separation of concerns:
• “preprocessing” for data prep
• “model” for training & inference logic
• task folders for domain‐specific pipelines
• Clear data‐code boundaries: data folders versus prog/model folders
• Use of CLI wrappers and shell scripts for reproducibility
Detail and Clarity
• Intermediate files (e.g., .npy, .h5ad) where relevant
• Both batch workflows (via run.sh scripts) and interactive ones (notebooks)