# vandijklab/cell2sentence architecture

> Architecture diagram of the GitHub repository vandijklab/cell2sentence, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vandijklab/cell2sentence
- Repository: https://github.com/vandijklab/cell2sentence

## Overview

##### Project type and purpose
- Cell2Sentence is a Python library and framework (not a web app or microservice) for converting single‑cell gene expression data into “cell sentences” and interfacing with Large Language Models (LLMs) to perform tasks like gene reconstruction, cell type prediction, and conditional cell generation.
- It provides core modules for data ingestion (`csdata`), model interaction (`csmodel`), prompt formatting (`prompt_formatter`), and task orchestration (`tasks`), packaged along with tutorials, tests, and documentation.

##### Core components
a. Data Ingestion & Preprocessing (csdata)
- Inputs: raw single‑cell files (`.csv`, `.h5ad`)
- Operations: loading, filtering, normalization, rank‑order transformation to produce cell sentences
- Output: structured data objects (e.g., pandas DataFrame, AnnData, or custom `CSData` objects)
b. Prompt Construction (prompt_formatter)
- Inputs: transformed cell sentences + JSON templates (`src/cell2sentence/prompts/*.json`)
- Operations: fill template placeholders, assemble natural‑language prompts for specific tasks
- Output: finalized text prompts ready for LLM consumption
c. Model Wrapper & Inference/Finetuning (csmodel)
- Dependencies: Huggingface Transformers (e.g., Pythia), optional Flash‑Attention for speed
- Operations: load pretrained C2S models or fine‑tune on user data (via PyTorch Trainer API), run inference to predict next gene tokens or cell types
- Output: token sequences, embeddings, probabilities, reconstructed expression vectors
d. Task Orchestration (tasks)
- Built‑in pipelines for:
- rank transformation & inverse reconstruction
- embedding extraction
- cell type prediction
- conditional cell generation
- multi‑cell and multi‑task workflows
- Coordinates calls to `csdata`, `prompt_formatter`, and `csmodel`
- Returns final results to the user (arrays, DataFrames, text)
e. Utilities & I/O (utils)
- Logging, caching, evaluation metrics, I/O helpers
f. External Services & Artifacts
- Huggingface Hub for model download/upload
- Optional Flash‑Attention C++ backend
- Tutorials/notebooks (Jupyter) for guided workflows
- Documentation (Sphinx) and CI/CD (GitHub Actions)

##### Architectural patterns & design principles
- Modular layering: clear separation between data, prompt logic, model interaction, and orchestration
- Adapter pattern: `CSModel` wraps any Huggingface model to expose consistent C2S‑specific API
- Template method pattern: `PromptFormatter` processes generic JSON templates for different tasks
- Pipeline orchestration: `tasks` module as higher‑order functions chaining modules
- Extensibility: JSON‑based prompts, plugin‑style tasks, easy to add new workflows

##### Technologies, frameworks, and other artifacts
- Python 3.8+
- Dependencies: `transformers`, `torch`, `scanpy/anndata` (for `.h5ad`), `pandas`, `json`, optional `flash‑attention`
- Packaging: `pyproject.toml`, `setup.cfg`, `Makefile`, Conda environment
- Testing: `pytest` with sample data under `src/cell2sentence/tests`
- Documentation: Sphinx config in `docs/source`, built via ReadTheDocs, CI pipeline in `.github/workflows`

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Core Components
    subgraph "Cell2Sentence Core" 
        direction TB
        subgraph "Data Ingestion & Preprocessing"
            direction TB
            csdata["csdata\n(load, filter, normalize)"]:::data
        end
        subgraph "Prompt Construction"
            direction TB
            prompt_formatter["prompt_formatter\n(fill JSON templates)"]:::prompt
            json_templates["prompts/*.json"]:::prompt
        end
        subgraph "Model Wrapper & Inference/Finetuning"
            direction TB
            csmodel["csmodel\n(load, fine-tune, inference)"]:::model
            flash["Flash-Attention (optional)"]:::external
        end
        subgraph "Task Orchestration"
            direction TB
            tasks["tasks\n(pipelines & orchestration)"]:::orchestration
        end
        subgraph "Utilities & I/O"
            direction TB
            utils["utils\n(logging, caching, I/O)"]:::util
        end
    end

    %% External & Artifacts
    subgraph "External & Ecosystem"
        direction TB
        hf["Huggingface Hub"]:::external
        tests["Testing Suite"]:::meta
        tutorials["Tutorials & Notebooks"]:::meta
        docs["Documentation (Sphinx)"]:::meta
        ci["CI/CD Pipeline"]:::meta
        packaging["Packaging & Config"]:::meta
    end

    %% Data Flow
    csdata -->|"cell sentences"| prompt_formatter
    prompt_formatter -->|"text prompts"| csmodel
    csmodel -->|"raw outputs"| tasks
    tasks -->|"results (DataFrames, text, arrays)"| UserOutput["User Outputs"]:::external

    %% Fine-tuning loop
    csdata -->|"dataset"| csmodel
    csmodel -.->|"optional integration"| flash
    csmodel -->|"upload/download"| hf
    tasks -->|"use utils"| utils

    %% Meta connections
    tests -->|tests modules| csdata
    tests -->|tests modules| prompt_formatter
    tests -->|tests modules| csmodel
    tests -->|tests modules| tasks
    tutorials -->|example workflows| csdata
    tutorials -->|example workflows| tasks
    docs -->|API docs| csdata
    docs -->|API docs| csmodel
    docs -->|API docs| prompt_formatter
    docs -->|API docs| tasks
    docs -->|API docs| utils
    ci -->|runs tests & docs| tests
    packaging -->|defines env & build| csdata
    packaging -->|defines env & build| csmodel

    %% Click Events
    click csdata "https://github.com/vandijklab/cell2sentence/blob/master/src/cell2sentence/csdata.py"
    click prompt_formatter "https://github.com/vandijklab/cell2sentence/blob/master/src/cell2sentence/prompt_formatter.py"
    click json_templates "https://github.com/vandijklab/cell2sentence/blob/master/src/cell2sentence/prompts/*.json"
    click csmodel "https://github.com/vandijklab/cell2sentence/blob/master/src/cell2sentence/csmodel.py"
    click tasks "https://github.com/vandijklab/cell2sentence/blob/master/src/cell2sentence/tasks.py"
    click utils "https://github.com/vandijklab/cell2sentence/blob/master/src/cell2sentence/utils.py"
    click tests "https://github.com/vandijklab/cell2sentence/tree/master/src/cell2sentence/tests/"
    click tutorials "https://github.com/vandijklab/cell2sentence/tree/master/tutorials/"
    click docs "https://github.com/vandijklab/cell2sentence/tree/master/docs/source/"
    click ci "https://github.com/vandijklab/cell2sentence/blob/master/.github/workflows/main.yml"
    click packaging "https://github.com/vandijklab/cell2sentence/blob/master/pyproject.toml"

    %% Styles
    classDef data fill:#b3cde3,stroke:#03396c,color:#03396c
    classDef prompt fill:#c7e9c0,stroke:#238b45,color:#238b45
    classDef model fill:#fdd0a2,stroke:#e34a33,color:#e34a33
    classDef orchestration fill:#dadaeb,stroke:#4a1486,color:#4a1486
    classDef util fill:#f2f0f7,stroke:#6a51a3,color:#6a51a3
    classDef external fill:#f0f0f0,stroke:#636363,color:#636363
    classDef meta fill:#fff7bc,stroke:#b15928,color:#b15928
```
