How bowang-lab/scgpt works
Project Type and Purpose
• scGPT is a Python package and toolkit for building and fine-tuning a foundation model for single-cell multi-omics data.
• It provides data-processing scripts, a databank module, preprocessing and tokenization layers, a model core (PyTorch), a training engine, examples, tutorials, and documentation.
File Structure
• Top-level directories:
• data/ – cellxgene data ingestion & preprocessing scripts (shell + Python)
• scgpt/ – main library, with subpackages for databank, tokenizer, model, tasks, utils, and training
• examples/ – sample scripts using the API (e.g., finetune_integration.py)
• docs/ & tutorials/ – documentation and Jupyter notebooks demonstrating use cases
• tests/ – unit tests validating each component
• Key configuration and packaging files: pyproject.toml, .readthedocs.yaml, LICENSE.
README for Additional Insights
• Core components:
• Pretrained model zoo and loading utilities (scgpt/utils/util.py)
• Optional dependencies: flash-attn, faiss, wandb, HuggingFace integration
• Online apps (Superbio.ai) that wrap the core library for inference services.
Main Components and Their Interactions
a. Data Ingestion & Preprocessing Subsystem
• data/cellxgene/*.sh and *.py scripts: download, partition, process raw single-cell data
• data_config.py, metainfo.json, cancer_list.txt, query_list.txt
• Outputs: processed count matrices and metadata files
b. Databank Module (scgpt/scbank)
• Data classes (data.py), Databank manager (databank.py), Monitor (monitor.py), Settings (setting.py)
• Abstracts loading, caching, indexing of large single-cell datasets (e.g., FAISS indexes)
c. Preprocessing & Tokenization
• scgpt/preprocess.py: transforms matrices into token sequences
• scgpt/tokenizer: default vocab JSONs + GeneTokenizer class
d. Model Core (scgpt/model)
• Base model and variants (model.py, generation_model.py, multiomic_model.py)
• Layers: DSBN (dsbn.py), gradient reversal (grad_reverse.py)
e. Training Engine & Task Layer
• scgpt/trainer.py: orchestrates training loops, callbacks
• scgpt/data_collator.py & data_sampler.py: batching strategies
• scgpt/loss.py: custom objective functions
• scgpt/tasks/: implementations for cell embedding, GRN inference, perturbation prediction, reference mapping
f. API, Examples & Deployment
• examples/finetune_integration.py: demonstrates API usage for fine-tuning
• Packaging (pyproject.toml) for PyPI distribution
• CLI wrappers or entrypoints (if any)
g. External Services & Tooling
• PyTorch (model training/inference)
• flash-attn (optional speedup)
• faiss (reference mapping index)
• wandb (logging/visualization)
• HuggingFace integration branch (future)
• Superbio.ai webapps for hosted inference
h. Documentation & Testing
• docs/: Sphinx configs and .rst files (architecture, tutorials, API references)
• tests/: unit tests for sampler, tokenizer, scbank, model
• Blue = core library modules (scgpt/…)
• Green = data processing & storage
• Orange = training & tasks
• Purple = external dependencies/services
• Raw data → Data Ingestion scripts → Databank → Preprocessing → Tokenization → Model → Trainer → Outputs
• Trainer → Wandb (logging)
• Trainer → faiss (indexing)
• Model/API → Superbio.ai (deployed service)
Architectural Patterns & Principles
• Modular layering: clear separation of data handling, preprocessing, modeling, and training
• Pipeline orchestration: sequential dataflow from raw matrices to model outputs
• Plug-in tasks: Task modules implement interchangeable learning objectives
• Extensibility: easy addition of new tokenizers, models, and tasks