How tencentailabhealthcare/scbert works
Main Components
a) Data Processing Layer:
• Input data flow starting with raw scRNA-seq data
• Preprocessing components (reference preprocess.py):
• NCBI Gene database validation
• Gene symbol revision
• Normalization (scanpy integration)
• Log transformation
b) Model Architecture Layer:
• Performer Transformer Architecture (from performer_pytorch/):
• Main Performer-LM component with its key parameters:
• Token embedding (num_tokens=7)
• Dimensional embedding (dim=200)
• Multi-head attention (heads=10)
• Encoder layers (depth=6)
• Reversible layers (indicated by reversible.py)
c) Training Pipeline:
• Pre-training phase (pretrain.py)
• Fine-tuning phase (finetune.py)
• Prediction pipeline (predict.py)
Flow and Relationships
• Data flow from preprocessing → model input
• Pre-trained model → Fine-tuning process
• Fine-tuned model → Prediction system
• Novel cell type detection branch (with threshold mechanism)
Additional Elements
• Component types
• Key parameters and thresholds
Technical Notes
Annotations for:
• Required dependencies (PyTorch, transformers, scanpy, etc.)
• Computational requirements
• Distributed training capabilities
• Memory considerations
Remember to indicate that this is a research tool and include appropriate attribution to Tencent AI Lab.