# biomap-research/scfoundation architecture

> Architecture diagram of the GitHub repository biomap-research/scfoundation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/biomap-research/scfoundation
- Repository: https://github.com/biomap-research/scfoundation

## Overview

##### Project Type and Purpose
- This is a research‐oriented, multi‐task machine‐learning repository centered on a large‐scale pretrained single‐cell transcriptomics foundation model (“scFoundation”).
- It provides pipelines for (a) pre‐training, (b) data preprocessing, (c) embedding inference, and (d) downstream tasks (DeepCDR, SCAD, GEARS, read‐depth enhancement, annotation, mapping, gene‐module inference, ablation studies, API examples).
- It is not a classical full‐stack web app; rather it is a modular ML toolkit with data, model code, scripts, notebooks, published weights, and CLI wrappers.

##### Key Components (to include
A. Data Sources & Stores
- Raw single‐cell data repository (preprocessing/data/original & Figshare links)
- Processed data (preprocessing/data/processing, SCAD/data/split_norm, DeepCDR/data, GEARS/data/demo)
- Published pretrained weights (model/pretrainmodels & DeepCDR/published)
- Task‐specific result folders (SCAD/results, enhancement outputs, annotation data)
B. Preprocessing Pipeline
- Entry point: preprocessing/scRNA_workflow.py & demo.sh
- Downloads & normalizes raw data → produces normalized gene expression matrices
C. Pre‐training & Embedding Generation
- Core model code (model/ folder) implementing xTrimoGene variants (mae_autobin, performer, transformer, reversible)
- Training script(s) (model/finetune_model.py, train with DeepSpeed/PyTorch Lightning)
- Embedding extraction (model/get_embedding.py, model/load.py)
- Output: multi‐million‐cell embeddings (50M‐0.1B‐res_embedding.npy)
D. Inference & API
- CLI example (apiexample/client.py, example.sh) calling new inference endpoint (https://aigp.biomap.com/)
- Local inference utilities in model/ or GEARS/inference.py
E. Downstream Task Modules
- DeepCDR folder: processes bulk data + embeddings → drug response prediction
- SCAD folder: single‐cell drug sensitivity prediction using embedding inputs
- GEARS folder: perturbation prediction; generates gene embeddings in training
- enhancement folder: read‐depth enhancement via embeddings
- annotation folder: cell‐type annotation pipelines
- mapping folder: organoid → in vivo mapping demos
- genemodule folder: infers gene modules from gene context embeddings
- ablation folder: comparative studies of model & loss variants
F. Utilities & Scripts
- Shell scripts (run.sh in each task folder) for reproducible experiments
- Jupyter notebooks for data exploration & plotting

##### Relationships & Data Flow
- Raw data → Preprocessing → Processed data
- Processed data + pretraining code → scFoundation weights + cell/gene embeddings
- Embeddings (and related processed data) feed into each downstream module
- Downstream modules produce results/figures and write to their results folders
- API client interacts with a remote inference service using the same embedding models
- Notebooks consume outputs at each stage for visualization

##### Architectural Patterns & Principles
- Modularization by task: each downstream experiment lives in its own folder with clear inputs, code, and outputs
- Reuse of core model code for both pretraining and inference across tasks
- Separation of concerns:
- “preprocessing” for data prep
- “model” for training & inference logic
- task folders for domain‐specific pipelines
- Clear data‐code boundaries: data folders versus prog/model folders
- Use of CLI wrappers and shell scripts for reproducibility

##### Detail and Clarity
- Intermediate files (e.g., .npy, .h5ad) where relevant
- Both batch workflows (via run.sh scripts) and interactive ones (notebooks)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Stores
    subgraph "Data Stores"
        style DataStores fill:none,stroke:none
        RawRepo["Raw Single-Cell Data\n(SCAD/data/original &\nSCAD/data/original_norm)"]:::data
        ProcData["Processed Data\n(preprocessing/data/processing)"]:::data
        SCADSplit["SCAD Split & Normalized\nEmbeddings\n(SCAD/data/split_norm)"]:::data
        DeepCDRData["DeepCDR Processed Data &\nEmbeddings\n(DeepCDR/data)"]:::data
        GEARSDemo["GEARS Demo Data\n(GEARS/data/demo.h5ad,\nGEARS/data/gene2go.pkl)"]:::data
        Embeddings["Cell/Gene Embeddings\n(50M-0.1B-res_embedding.npy)"]:::data
        PretrainWeights["scFoundation Pretrained Weights\n(model/pretrainmodels)"]:::data
        DeepCDRWeights["DeepCDR Published Weights\n(DeepCDR/published)"]:::data
        SCADResults["SCAD Results\n(SCAD/results)"]:::data
        EnhanceOut["Enhancement Outputs\n(enhancement/*.npy)"]:::data
        AnnotationData["Annotation Data\n(annotation/annotation_data.zip)"]:::data
        AblationNB["Ablation Notebooks\n(ablation/*.ipynb)"]:::data
    end

    %% Compute Modules
    subgraph "Compute Modules"
        style Compute fill:none,stroke:none
        Preprocess["Preprocessing Pipeline\n(preprocessing/scRNA_workflow.py,\n demo.sh)"]:::compute
        Pretrain["Pre-training & Embedding Gen\n(model/finetune_model.py)"]:::compute
        GetEmbed["Embedding Extraction\n(model/get_embedding.py)"]:::compute
        LoadModel["Model Loader\n(model/load.py)"]:::compute
        APIClient["API Client\n(apiexample/client.py,\n example.sh)"]:::external
    end

    %% Code Libraries
    subgraph "Code Libraries"
        style CodeLibs fill:none,stroke:none
        ModelCore["Core Model Definitions\n(model/pretrainmodels/*)"]:::code
        GEARSInfer["GEARS Inference Utils\n(GEARS/gears/inference.py)"]:::code
    end

    %% Downstream Tasks
    subgraph "Downstream Task Pipelines"
        style Tasks fill:none,stroke:none
        DeepCDRMod["DeepCDR Module\n(DeepCDR/)"]:::compute
        SCADMod["SCAD Module\n(SCAD/)"]:::compute
        GEARSMod["GEARS Module\n(GEARS/)"]:::compute
        EnhanceMod["Read-Depth Enhancement\n(enhancement/)"]:::compute
        AnnotMod["Annotation Pipeline\n(annotation/)"]:::compute
        MappingMod["Mapping Demo\n(mapping/)"]:::compute
        GeneMod["Gene Module Inference\n(genemodule/)"]:::compute
        AblationMod["Ablation Studies\n(ablation/)"]:::compute
    end

    %% Utilities & Notebooks
    subgraph "Utilities & Notebooks"
        style Utils fill:none,stroke:none
        ShellScripts["Shell Scripts\n(run.sh, demo.sh, etc.)"]:::utility
        Notebooks["Jupyter Notebooks\n(various .ipynb)"]:::utility
    end

    %% Data Flow
    RawRepo -->|normalized matrices| Preprocess
    Preprocess -->|processed data| ProcData
    ProcData --> Pretrain
    ModelCore -.-> Pretrain
    Pretrain -->|scFoundation weights| PretrainWeights
    Pretrain -->|raw embeddings| Embeddings
    GetEmbed --> Embeddings
    LoadModel --> Embeddings
    ModelCore -.-> GetEmbed
    ModelCore -.-> LoadModel

    Embeddings --> DeepCDRMod
    Embeddings --> SCADMod
    Embeddings --> GEARSMod
    Embeddings --> EnhanceMod
    Embeddings --> AnnotMod
    Embeddings --> MappingMod
    Embeddings --> GeneMod
    Embeddings --> AblationMod

    DeepCDRMod -->|results| DeepCDRData
    SCADMod -->|results| SCADResults
    GEARSMod -->|demo outputs| GEARSDemo
    EnhanceMod -->|enhanced data| EnhanceOut
    AnnotMod -->|annotations| AnnotationData
    MappingMod -->|mapping results| AnnotationData
    GeneMod -->|inferred modules| GeneMod
    AblationMod -->|study notebooks| AblationNB

    APIClient -->|inference request| API["Inference Service\n(https://aigp.biomap.com/)"]:::external
    API -->|predictions| Embeddings

    ShellScripts --> Preprocess
    ShellScripts --> Pretrain
    ShellScripts --> DeepCDRMod
    ShellScripts --> SCADMod
    ShellScripts --> GEARSMod
    ShellScripts --> EnhanceMod
    ShellScripts --> AnnotMod
    ShellScripts --> MappingMod
    ShellScripts --> GeneMod
    ShellScripts --> AblationMod
    Notebooks --> ProcData
    Notebooks --> Embeddings
    Notebooks --> DeepCDRMod
    Notebooks --> SCADMod
    Notebooks --> GEARSMod
    Notebooks --> EnhanceMod
    Notebooks --> AnnotMod
    Notebooks --> MappingMod
    Notebooks --> GeneMod
    Notebooks --> AblationMod

    %% Click Events
    click RawRepo "https://github.com/biomap-research/scfoundation/tree/main/SCAD/data/original"
    click ProcData "https://github.com/biomap-research/scfoundation/tree/main/SCAD/data/processing"
    click SCADSplit "https://github.com/biomap-research/scfoundation/tree/main/SCAD/data/split_norm"
    click DeepCDRData "https://github.com/biomap-research/scfoundation/tree/main/DeepCDR/data"
    click GEARSDemo "https://github.com/biomap-research/scfoundation/blob/main/GEARS/data/demo.h5ad"
    click PretrainWeights "https://github.com/biomap-research/scfoundation/tree/main/model/pretrainmodels"
    click DeepCDRWeights "https://github.com/biomap-research/scfoundation/tree/main/DeepCDR/published"
    click SCADResults "https://github.com/biomap-research/scfoundation/tree/main/SCAD/results"
    click EnhanceOut "https://github.com/biomap-research/scfoundation/tree/main/enhancement"
    click AnnotationData "https://github.com/biomap-research/scfoundation/blob/main/annotation/annotation_data.zip"
    click AblationNB "https://github.com/biomap-research/scfoundation/tree/main/ablation"
    click Preprocess "https://github.com/biomap-research/scfoundation/blob/main/preprocessing/scRNA_workflow.py"
    click ModelCore "https://github.com/biomap-research/scfoundation/tree/main/model/pretrainmodels"
    click GetEmbed "https://github.com/biomap-research/scfoundation/blob/main/model/get_embedding.py"
    click LoadModel "https://github.com/biomap-research/scfoundation/blob/main/model/load.py"
    click APIClient "https://github.com/biomap-research/scfoundation/blob/main/apiexample/client.py"
    click GEARSInfer "https://github.com/biomap-research/scfoundation/blob/main/GEARS/gears/inference.py"
    click DeepCDRMod "https://github.com/biomap-research/scfoundation/tree/main/DeepCDR/"
    click SCADMod "https://github.com/biomap-research/scfoundation/tree/main/SCAD/"
    click GEARSMod "https://github.com/biomap-research/scfoundation/tree/main/GEARS/"
    click EnhanceMod "https://github.com/biomap-research/scfoundation/tree/main/enhancement/"
    click AnnotMod "https://github.com/biomap-research/scfoundation/tree/main/annotation/"
    click MappingMod "https://github.com/biomap-research/scfoundation/tree/main/mapping/"
    click GeneMod "https://github.com/biomap-research/scfoundation/tree/main/genemodule/"
    click AblationMod "https://github.com/biomap-research/scfoundation/tree/main/ablation/"
    click ShellScripts "https://github.com/biomap-research/scfoundation/blob/main/SCAD/run.sh"
    click Notebooks "https://github.com/biomap-research/scfoundation/blob/main/preprocessing/demo.ipynb"

    %% Styles
    classDef data fill:#a8ddb5,stroke:#31a354,color:#000;
    classDef compute fill:#9ecae1,stroke:#3182bd,color:#000;
    classDef code stroke-dasharray: 5 5,fill:#f0f0f0,stroke:#636363,color:#000;
    classDef external fill:#fdae6b,stroke:#e6550d,color:#000;
    classDef utility fill:#fff7bc,stroke:#fec44f,color:#000;
```
