# epigen/cellwhisperer architecture

> Architecture diagram of the GitHub repository epigen/cellwhisperer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/epigen/cellwhisperer
- Repository: https://github.com/epigen/cellwhisperer

## Overview

1. **Overall shape**
   CellWhisperer is not a single app; it is a repo that combines model code, multiple Snakemake pipelines, dataset curation, evaluation/figure generation, and deployment assets. The main product is a multimodal system for scRNA-seq exploration: transcriptome embeddings plus a chat interface over CELLxGENE.

2. **Top-level execution boundaries**
   The repository separates three major execution modes:
   - **offline data/model pipelines** driven by Snakemake under `src/`
   - **Python package runtime** under `src/cellwhisperer/`
   - **deployment/hosting** under `hosting/`, `Dockerfile`, and `modules/cellxgene`
   This matters for architecture because most outputs are generated artifacts in `resources/` and `results/`, not source-controlled code.

3. **Core embedding package**
   `src/cellwhisperer/` is the main Python library. It contains:
   - model definitions for the joint embedding system
   - dataset wrappers for training and inference
   - loss functions and contrastive objectives
   - training entrypoints via PyTorch Lightning
   - utilities for inference, model I/O, and processing
   This is the central runtime layer used both for training and for generating embeddings for new datasets.

4. **Embedding model variants**
   The package supports multiple transcriptome backbones and variants, including `geneformer_model.py`, `scgpt_model.py`, and `uce_model.py`, plus `frozen_model.py` and joint embedding orchestration. These are boundary points where the repo adapts external foundation models into a common CellWhisperer interface.

5. **Training data engineering**
   The training corpora are built from several dataset-specific pipelines under `src/datasets/` such as ARCHS4/GEO, CELLxGENE Census, Tabula Sapiens, ImmGen, pancreas, bowel disease, and others. Each dataset family has its own Snakemake rules and scripts for downloading, filtering, harmonizing metadata, and producing H5AD-based training inputs.

6. **Natural-language annotation pipeline**
   `src/pre_training_processing/` transforms structured single-cell metadata into captions and annotations used for multimodal training. It includes scripts for annotation processing and dataset integration, plus prompts and sample request/response pairs. The README indicates this stage may use GPT-4 when available, or a local Qwen model fallback, so annotation generation is an explicit NLP subpipeline rather than part of the model itself.

7. **Joint transcriptome–text training**
   The contrastive/joint embedding training workflow lives in the `cellwhisperer` package and is configured through YAML files such as `src/cellwhisperer_clip_v1.yaml` and the v2 variants. Training is run with the LightningCLI-style `cellwhisperer fit ...` workflow. Losses and discriminators under `src/cellwhisperer/jointemb/loss/` implement the alignment objective between transcriptome and text embeddings.

8. **Chat-model pipeline**
   `src/llava/` is a separate training pipeline for the CellWhisperer chat model, built on a modified LLaVA stack. It contains dataset-generation scripts, prompt templates, request/response examples, and Snakemake rules for training. This is architecturally distinct from the embedding model: one subsystem produces vector embeddings, the other powers conversational generation over cell data.

9. **User dataset preprocessing for CELLxGENE**
   `src/cellxgene_preprocessing/` is the pipeline that turns a user’s H5AD dataset into a CELLxGENE-ready dataset with CellWhisperer features. It reads raw counts, generates embeddings, annotates clusters, and outputs a processed H5AD used by the browser. This is the main boundary for “bring your own dataset” support.

10. **Analysis and paper reproduction**
    `src/figures/` and `src/ablation_study/` are downstream analytical pipelines for reproducing manuscript figures, ablations, and validation experiments. They consume outputs from the training and preprocessing stages, often along with precomputed artifacts, and produce the paper’s plots and tables. These directories are analysis consumers rather than production runtime components.

11. **Serving and deployment**
    `hosting/` contains the public website and deployment setup, including static pages, nginx configuration, docker compose files, and a dataset-processing Docker setup. `src/model_server/` is a lightweight server entrypoint for model-backed functionality. The deployment path also integrates the modified CELLxGENE submodule so the browser can call CellWhisperer-backed services.

12. **External dependencies and submodules**
    `modules/` holds git submodules for modified upstream projects: **CELLxGENE**, **LLaVA**, **Geneformer**, and **UCE**. These are not merely vendored copies; they form explicit architectural seams where the repo extends external ML and browser infrastructure. The repo also depends on external services for hosted AI inference by default, plus optional OpenAI, Hugging Face, and project-hosted dataset/model downloads.

13. **Shared infrastructure**
    `src/shared/` centralizes rules and utilities reused across pipelines: model downloads, gene sets, GSVA, sample weights, annotation representations, and transcriptome processing. This layer reduces duplication across dataset prep, training, and evaluation, and is the main cross-cutting dependency among the Snakemake workflows.

Main entry points: [`src/Snakefile`](https://github.com/epigen/cellwhisperer/tree/main/src/Snakefile) (Main pipeline), [`src/cellwhisperer/jointemb`](https://github.com/epigen/cellwhisperer/tree/main/src/cellwhisperer/jointemb) (Joint emb), [`hosting`](https://github.com/epigen/cellwhisperer/tree/main/hosting) (Hosting), [`src/cellwhisperer`](https://github.com/epigen/cellwhisperer/tree/main/src/cellwhisperer) (CW package)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_offline["Offline pipelines"]
  node_node_master_snakemake["Main pipeline<br/>snakemake entry"]
  node_node_shared["Shared core<br/>common rules"]
  node_node_datasets["Training data<br/>dataset pipelines"]
  node_node_prep_ann["Annotations<br/>NLP prep"]
  node_node_cellxgene_prep["CXG prep<br/>dataset preprocessing"]
  node_node_llava["Chat pipeline<br/>LLaVA training"]
  node_node_figures["Figures<br/>analysis pipeline"]
  node_node_ablation["Ablations<br/>analysis pipeline"]
end

subgraph group_group_runtime["Runtime library"]
  node_node_cellwhisperer_pkg["CW package<br/>python package"]
  node_node_jointemb["Joint emb<br/>embedding subsystem"]
  node_node_models["Model backbones<br/>encoder adapters"]
  node_node_losses["Objectives<br/>losses"]
  node_node_utils["Inference utils<br/>runtime helpers"]
  node_node_validation["Validation<br/>eval helpers"]
end

subgraph group_group_serving["Serving and deployment"]
  node_node_server(("Model server<br/>api entrypoint<br/>[main.py]"))
  node_node_hosting["Hosting<br/>deployment assets"]
end

subgraph group_group_external["External seams"]
  node_node_modules_cellxgene["CELLxGENE<br/>submodule"]
  node_node_modules_llava["LLaVA<br/>submodule"]
  node_node_modules_geneformer["Geneformer<br/>submodule"]
  node_node_modules_uce["UCE<br/>submodule"]
end

node_node_repo_root{{"CellWhisperer"}}

node_node_repo_root -->|"orchestrates"| node_node_master_snakemake
node_node_repo_root -->|"contains"| node_node_cellwhisperer_pkg
node_node_repo_root -->|"deploys via"| node_node_hosting
node_node_master_snakemake -->|"reuses"| node_node_shared
node_node_master_snakemake -->|"drives"| node_node_datasets
node_node_master_snakemake -->|"drives"| node_node_prep_ann
node_node_master_snakemake -->|"drives"| node_node_cellxgene_prep
node_node_master_snakemake -->|"drives"| node_node_llava
node_node_master_snakemake -->|"drives"| node_node_figures
node_node_master_snakemake -->|"drives"| node_node_ablation
node_node_shared -->|"supports"| node_node_datasets
node_node_shared -->|"supports"| node_node_figures
node_node_prep_ann -->|"feeds"| node_node_cellwhisperer_pkg
node_node_datasets -->|"feeds"| node_node_cellwhisperer_pkg
node_node_cellxgene_prep -->|"feeds"| node_node_hosting
node_node_llava -->|"extends"| node_node_modules_llava
node_node_cellwhisperer_pkg -->|"centers on"| node_node_jointemb
node_node_jointemb -->|"adapts"| node_node_models
node_node_jointemb -->|"optimizes with"| node_node_losses
node_node_jointemb -->|"uses"| node_node_utils
node_node_jointemb -->|"evaluates with"| node_node_validation
node_node_models -->|"wraps"| node_node_modules_geneformer
node_node_models -->|"wraps"| node_node_modules_uce
node_node_hosting -->|"includes"| node_node_server
node_node_hosting -->|"integrates"| node_node_modules_cellxgene

click node_node_master_snakemake "https://github.com/epigen/cellwhisperer/tree/main/src/Snakefile"
click node_node_shared "https://github.com/epigen/cellwhisperer/tree/main/src/shared"
click node_node_datasets "https://github.com/epigen/cellwhisperer/tree/main/src/datasets"
click node_node_prep_ann "https://github.com/epigen/cellwhisperer/tree/main/src/pre_training_processing"
click node_node_cellxgene_prep "https://github.com/epigen/cellwhisperer/tree/main/src/cellxgene_preprocessing"
click node_node_llava "https://github.com/epigen/cellwhisperer/tree/main/src/llava"
click node_node_figures "https://github.com/epigen/cellwhisperer/tree/main/src/figures"
click node_node_ablation "https://github.com/epigen/cellwhisperer/tree/main/src/ablation_study"
click node_node_cellwhisperer_pkg "https://github.com/epigen/cellwhisperer/tree/main/src/cellwhisperer"
click node_node_jointemb "https://github.com/epigen/cellwhisperer/tree/main/src/cellwhisperer/jointemb"
click node_node_models "https://github.com/epigen/cellwhisperer/tree/main/src/cellwhisperer/jointemb"
click node_node_losses "https://github.com/epigen/cellwhisperer/tree/main/src/cellwhisperer/jointemb/loss"
click node_node_utils "https://github.com/epigen/cellwhisperer/tree/main/src/cellwhisperer/utils"
click node_node_validation "https://github.com/epigen/cellwhisperer/tree/main/src/cellwhisperer/validation"
click node_node_server "https://github.com/epigen/cellwhisperer/blob/main/src/model_server/main.py"
click node_node_hosting "https://github.com/epigen/cellwhisperer/tree/main/hosting"
click node_node_modules_cellxgene "https://github.com/epigen/cellwhisperer/tree/main/modules/cellxgene"
click node_node_modules_llava "https://github.com/epigen/cellwhisperer/tree/main/modules/LLaVA"
click node_node_modules_geneformer "https://github.com/epigen/cellwhisperer/tree/main/modules/Geneformer"
click node_node_modules_uce "https://github.com/epigen/cellwhisperer/tree/main/modules/UCE"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_master_snakemake,node_node_shared,node_node_datasets,node_node_prep_ann,node_node_cellxgene_prep,node_node_llava,node_node_figures,node_node_ablation toneBlue
class node_node_cellwhisperer_pkg,node_node_jointemb,node_node_models,node_node_losses,node_node_utils,node_node_validation toneAmber
class node_node_server,node_node_hosting toneMint
class node_node_modules_cellxgene,node_node_modules_llava,node_node_modules_geneformer,node_node_modules_uce toneRose
class node_node_repo_root toneNeutral
```

## Components

### Offline pipelines

Snakemake-driven data preparation, annotation, and analysis workflows.

- [Main pipeline](https://github.com/epigen/cellwhisperer/tree/main/src/Snakefile): snakemake entry · Top-level orchestration for repository-wide offline workflows. · `src/Snakefile`
- [Shared core](https://github.com/epigen/cellwhisperer/tree/main/src/shared): common rules · Reusable processing, model-download, gene-set, and weighting logic shared across pipelines. · `src/shared`
- [Training data](https://github.com/epigen/cellwhisperer/tree/main/src/datasets): dataset pipelines · Family of dataset-specific Snakemake workflows for building training corpora. · `src/datasets`
- [Annotations](https://github.com/epigen/cellwhisperer/tree/main/src/pre_training_processing): NLP prep · Transforms structured metadata into captions and annotations for multimodal training. · `src/pre_training_processing`
- [CXG prep](https://github.com/epigen/cellwhisperer/tree/main/src/cellxgene_preprocessing): dataset preprocessing · Converts user H5AD data into CELLxGENE-ready datasets with CellWhisperer features. · `src/cellxgene_preprocessing`
- [Chat pipeline](https://github.com/epigen/cellwhisperer/tree/main/src/llava): LLaVA training · Separate conversational model pipeline for cell-data question answering and evaluation. · `src/llava`
- [Figures](https://github.com/epigen/cellwhisperer/tree/main/src/figures): analysis pipeline · Manuscript figure generation, validation analysis, and downstream plotting workflows. · `src/figures`
- [Ablations](https://github.com/epigen/cellwhisperer/tree/main/src/ablation_study): analysis pipeline · Experimental variants and comparisons used for model ablation studies. · `src/ablation_study`

### Runtime library

Core Python package for joint embeddings, inference, and training.

- [CW package](https://github.com/epigen/cellwhisperer/tree/main/src/cellwhisperer): python package · Main runtime library for joint embeddings, training, inference, and model I/O. · `src/cellwhisperer`
- [Joint emb](https://github.com/epigen/cellwhisperer/tree/main/src/cellwhisperer/jointemb): embedding subsystem · Common interface for transcriptome-text alignment and model orchestration. · `src/cellwhisperer/jointemb`
- [Model backbones](https://github.com/epigen/cellwhisperer/tree/main/src/cellwhisperer/jointemb): encoder adapters · Adapters for Geneformer, scGPT, UCE, and frozen backbone variants. · `src/cellwhisperer/jointemb`
- [Objectives](https://github.com/epigen/cellwhisperer/tree/main/src/cellwhisperer/jointemb/loss): losses · Contrastive and discriminator components implementing alignment objectives. · `src/cellwhisperer/jointemb/loss`
- [Inference utils](https://github.com/epigen/cellwhisperer/tree/main/src/cellwhisperer/utils): runtime helpers · Shared inference, processing, and model serialization helpers. · `src/cellwhisperer/utils`
- [Validation](https://github.com/epigen/cellwhisperer/tree/main/src/cellwhisperer/validation): eval helpers · Integration and zero-shot evaluation utilities within the package and repo validation layer. · `src/cellwhisperer/validation`

### Serving and deployment

Hosting, browser integration, and model server entrypoints.

- [Model server](https://github.com/epigen/cellwhisperer/blob/main/src/model_server/main.py): api entrypoint · Lightweight service entrypoint for model-backed functionality. · `src/model_server/main.py`
- [Hosting](https://github.com/epigen/cellwhisperer/tree/main/hosting): deployment assets · Website, nginx, docker compose, and dataset-processing deployment setup. · `hosting`

### External seams

Submodules that extend or adapt upstream projects.

- [CELLxGENE](https://github.com/epigen/cellwhisperer/tree/main/modules/cellxgene): submodule · Modified browser and dataset exploration frontend integration seam. · `modules/cellxgene`
- [LLaVA](https://github.com/epigen/cellwhisperer/tree/main/modules/LLaVA): submodule · Upstream multimodal generation stack adapted for the chat pipeline. · `modules/LLaVA`
- [Geneformer](https://github.com/epigen/cellwhisperer/tree/main/modules/Geneformer): submodule · Upstream transcriptome foundation model adapted as an embedding backbone. · `modules/Geneformer`
- [UCE](https://github.com/epigen/cellwhisperer/tree/main/modules/UCE): submodule · Upstream transcriptome foundation model adapted as an embedding backbone. · `modules/UCE`

### Other components

- **CellWhisperer**: repo · Multimodal single-cell exploration system combining embedding models, chat workflows, pipelines, and deployment assets.

## Connections

- CellWhisperer → Main pipeline: orchestrates · Repository-wide offline workflows start here.
- CellWhisperer → CW package: contains · Core Python library lives under the package path.
- CellWhisperer → Hosting: deploys via · Hosting assets provide browser and service deployment.
- Main pipeline → Shared core: reuses · Common rules and helpers are imported across workflows.
- Main pipeline → Training data: drives · Dataset construction pipelines feed training inputs.
- Main pipeline → Annotations: drives · Annotation preprocessing contributes multimodal captions.
- Main pipeline → CXG prep: drives · Prepares user datasets for browser consumption.
- Main pipeline → Chat pipeline: drives · Chat-model dataset generation and training are pipeline-managed.
- Main pipeline → Figures: drives · Figure and analysis artifacts are generated from pipeline outputs.
- Main pipeline → Ablations: drives · Ablation experiments are executed as offline workflows.
- Shared core → Training data: supports · Dataset workflows depend on shared processing and downloads.
- Shared core → Figures: supports · Analysis notebooks and rules reuse shared utilities.
- Annotations → CW package: feeds · Generated annotations are used in multimodal training.
- Training data → CW package: feeds · Prepared corpora become training and inference inputs.
- CXG prep → Hosting: feeds · Processed H5AD artifacts are consumed by hosted browser workflows.
- Chat pipeline → LLaVA: extends · Chat pipeline is built on a modified LLaVA stack.
- CW package → Joint emb: centers on · Joint embedding subsystem is the main runtime core.
- Joint emb → Model backbones: adapts · Backbone-specific adapters normalize external encoders.
- Joint emb → Objectives: optimizes with · Alignment objectives train the shared embedding space.
- Joint emb → Inference utils: uses · Inference and model I/O helpers support training and serving.
- Joint emb → Validation: evaluates with · Package-level validation checks retrieval and annotation behavior.
- Model backbones → Geneformer: wraps · Geneformer backbone is integrated through an adapter.
- Model backbones → UCE: wraps · UCE backbone is integrated through an adapter.
- Hosting → Model server: includes · Deployment exposes the model server as a service component.
- Hosting → CELLxGENE: integrates · Hosted browser stack uses the modified CELLxGENE seam.