# biomed-ai/cellfm architecture

> Architecture diagram of the GitHub repository biomed-ai/cellfm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/biomed-ai/cellfm
- Repository: https://github.com/biomed-ai/cellfm

## Overview

##### Project Type and Purpose
- This is a machine-learning research codebase implementing “CellFM” (a single-cell foundation model) using MindSpore (with optional PyTorch) for pretraining and fine-tuning on transcriptomics data.
- It provides data-processing pipelines, core model definitions, training orchestration (single and multi-device), plus tutorials demonstrating downstream tasks (cell annotation, perturbation, batch integration, gene function prediction, etc.).

##### File-Structure Analysis
- Top-level directories and scripts
- train.py, 1node_train.sh, scheduler_sc100m.sh, worker_sc100m.sh: orchestration of training jobs (single vs. multi-device).
- config.py: central hyperparameter and path configurations.
- data_process.py, get_gene_emb.py: data cleaning, feature extraction.
- model.py, lora.py: model architecture (CellFM) and LoRA adapter.
- loss_function.py, metrics.py, earlystop.py, retention.py: training utilities (loss, performance metrics, early stopping, checkpoint retention).
- csv/: raw data files (gene info, mappings).
- tutorials/: example notebooks and sub-packages for downstream tasks.
- utils.py: shared utility functions.
- Configuration files: none external except optional YAMLs in tutorials.
- Deployment scripts: shell scripts for single-node multi-device training.

##### README Insights
- Describes installation (MindSpore 2.2.10, scanpy, scib; optional gears and torch).
- Workflow: preprocess h5/h5ad → train.py (single device) or 1node_train.sh (8 devices) → tutorials.
- External dataset source: Zenodo and HuggingFace.

##### Main Components and Interactions
a. Data Ingestion & Preprocessing
- Sources: CSV files, H5/H5AD datasets (Zenodo).
- Modules: data_process.py, get_gene_emb.py.
b. Model Definition
- Core: model.py implements transformer-based CellFM.
- Adapters: lora.py to inject LoRA low-rank modules.
c. Training Engine
- Orchestrator: train.py loads config, dataset, model, optimizer, scheduler.
- Utilities: loss_function.py, metrics.py, earlystop.py, retention.py govern training loop.
- Multi-device: 1node_train.sh launches scheduler_sc100m.sh + worker_sc100m.sh across NPUs.
d. Evaluation & Checkpointing
- Evaluation: metrics.py plus tutorial scripts.
- Checkpoint retention: retention.py enforces best-model retention policy.
e. Tutorials & Extensions
- Each tutorial folder contains task-specific code, config YAMLs, and setups for downstream tasks.

##### Architectural Patterns
- Modular ML pipeline: separation of data, model, training logic, and utilities.
- Adapter pattern: LoRA code injected at runtime for parameter-efficient fine-tuning.
- Script-based orchestration: shell scripts manage distributed resource scheduling.
- Extensible tutorial ecosystem: independent sub-packages illustrating use-cases.

##### Final Advice
- Be exhaustive: include every directory involved in model training and extension.
- Use layering: place core pipeline at center, orchestration on one side, tutorials on the other, and external services at the periphery.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Data Sources
    subgraph "External Services" 
        direction TB
        Zenodo["Zenodo"]:::external
        HF["HuggingFace"]:::external
    end

    %% Raw Data Storage
    csv["csv/ (Raw Data)"]:::storage
    click csv "https://github.com/biomed-ai/cellfm/tree/main/csv/"

    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing" 
        direction TB
        DP["data_process.py"]:::core
        click DP "https://github.com/biomed-ai/cellfm/blob/main/data_process.py"
        GE["get_gene_emb.py"]:::core
        click GE "https://github.com/biomed-ai/cellfm/blob/main/get_gene_emb.py"
    end

    %% Configuration
    CFG["config.py"]:::orchestration
    click CFG "https://github.com/biomed-ai/cellfm/blob/main/config.py"

    %% Model Definition
    subgraph "Model Definition" 
        direction TB
        M["model.py"]:::core
        click M "https://github.com/biomed-ai/cellfm/blob/main/model.py"
        ATT["attention.py"]:::core
        click ATT "https://github.com/biomed-ai/cellfm/blob/main/attention.py"
        LORA["lora.py"]:::core
        click LORA "https://github.com/biomed-ai/cellfm/blob/main/lora.py"
    end

    %% Training Orchestrator
    TO["train.py"]:::orchestration
    click TO "https://github.com/biomed-ai/cellfm/blob/main/train.py"

    %% Distributed Training
    subgraph "Distributed Training" 
        direction TB
        LAUNCH["1node_train.sh"]:::orchestration
        click LAUNCH "https://github.com/biomed-ai/cellfm/blob/main/1node_train.sh"
        SCH["scheduler_sc100m.sh"]:::orchestration
        click SCH "https://github.com/biomed-ai/cellfm/blob/main/scheduler_sc100m.sh"
        WKR["worker_sc100m.sh"]:::orchestration
        click WKR "https://github.com/biomed-ai/cellfm/blob/main/worker_sc100m.sh"
    end

    %% Training Utilities
    subgraph "Training Utilities" 
        direction TB
        LOSS["loss_function.py"]:::utility
        click LOSS "https://github.com/biomed-ai/cellfm/blob/main/loss_function.py"
        MET["metrics.py"]:::utility
        click MET "https://github.com/biomed-ai/cellfm/blob/main/metrics.py"
        ES["earlystop.py"]:::utility
        click ES "https://github.com/biomed-ai/cellfm/blob/main/earlystop.py"
        RET["retention.py"]:::utility
        click RET "https://github.com/biomed-ai/cellfm/blob/main/retention.py"
    end

    %% Shared Helpers
    UT["utils.py"]:::utility
    click UT "https://github.com/biomed-ai/cellfm/blob/main/utils.py"

    %% Tutorials Ecosystem
    subgraph "Tutorial Ecosystem" 
        direction TB
        TI1["tutorials/BatchIntegration/"]:::tutorial
        click TI1 "https://github.com/biomed-ai/cellfm/tree/main/tutorials/BatchIntegration/"
        TI2["tutorials/CellAnnotation/"]:::tutorial
        click TI2 "https://github.com/biomed-ai/cellfm/tree/main/tutorials/CellAnnotation/"
        TI3["tutorials/ChemicalPerturbation/"]:::tutorial
        click TI3 "https://github.com/biomed-ai/cellfm/tree/main/tutorials/ChemicalPerturbation/"
        TI4["tutorials/Perturbation/"]:::tutorial
        click TI4 "https://github.com/biomed-ai/cellfm/tree/main/tutorials/Perturbation/"
        TI5["tutorials/*.ipynb"]:::tutorial
        click TI5 "https://github.com/biomed-ai/cellfm/tree/main/tutorials/"
    end

    %% Data Flow
    Zenodo --> DP
    HF --> DP
    DP --> GE -->|"features"| M
    csv --> DP

    CFG --> TO
    GE --> TO
    M --> TO
    LORA --> TO
    UT --> TO

    TO --> LOSS --> TO
    TO --> MET --> TO
    TO --> ES --> TO
    TO --> RET --> TO

    TO --> LAUNCH --> SCH --> WKR
    WKR --> TO

    %% Tutorials reuse core
    TI1 --> DP
    TI1 --> M
    TI1 -->LOSS
    TI2 --> DP
    TI2 --> M
    TI2 -->MET
    TI3 --> DP
    TI3 --> M
    TI4 --> DP
    TI4 --> M
    TI5 --> DP
    TI5 --> M

    %% Legend
    subgraph "Legend"
        direction LR
        C1["Core Components"]:::core
        C2["Orchestration"]:::orchestration
        C3["Utilities"]:::utility
        C4["Tutorials"]:::tutorial
        C5["External/Storage"]:::external
    end

    classDef core fill:#AED6F1,stroke:#21618C,color:#000;
    classDef orchestration fill:#A9DFBF,stroke:#1D8348,color:#000;
    classDef utility fill:#F9E79F,stroke:#B7950B,color:#000;
    classDef tutorial fill:#D5D8DC,stroke:#5D6D7E,color:#000;
    classDef external fill:#F5B7B1,stroke:#C0392B,color:#000;
    classDef storage fill:#D7BDE2,stroke:#7D3C98,color:#000;
```
