# arcinstitute/state architecture

> Architecture diagram of the GitHub repository arcinstitute/state, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arcinstitute/state
- Repository: https://github.com/arcinstitute/state

## Overview

##### Project Type and Purpose
- This is an open-source, command-line driven machine-learning toolkit (“State”) for:
a. Learning cell-state embeddings (the “emb” subcommand)
b. Training and running state-transition predictive models (the “tx” subcommand)
- Users configure experiments via TOML files; the system loads single-cell datasets in h5ad format, preprocesses them, trains PyTorch Lightning models, and runs inference or builds/query vector databases.

##### File‐Structure Overview
- Top level:
- README, LICENSE, singularity.def, scripts, examples, assets
- pyproject.toml, .github (CI), .gitmodules (vector DB submodule)
- src/state:
- __main__.py: entry point for `state` CLI
- _cli/: command‐group definitions for `emb` and `tx` (fit, transform, query, preprocess, train, infer, predict)
- configs/: YAML templates for data, models, training, wandb
- emb/: embedding pipeline (loader.py, inference.py, finetune_decoder.py, train/trainer.py, vectordb.py)
- tx/: transition pipeline (data preprocessing, dataset classes, many model modules, PyTorch Lightning tasks)

##### Key Components
A. CLI Frontend
- Single binary “state” → dispatches to two command groups: emb and tx
B. Configuration Layer
- Loads CLI flags + TOML configs → merges with YAML defaults
C. Data Layer
- Reads raw h5ad files (anndata) → preprocess (normalize, log1p, HVG selection) → outputs processed h5ad or control templates
D. Embedding Subsystem (emb)
- Data Loader → Embedding Model (flash_transformer, decoder) → Trainer (PyTorch Lightning) → Checkpoints
- Inference/Transform: load checkpoint → generate embeddings → optional vector database operations
- VectorDB Builder & Query: uses LanceDB backend via vectordb.py
E. Transition Subsystem (tx)
- Preprocessing commands (preprocess_train, preprocess_infer)
- Dataset classes (scgpt_perturbation_dataset.py)
- Model Zoo: CPA, SCVI, SCGPT, decoder_only, etc. (each in its own module)
- LightningModel wrappers and Trainer → Checkpoints
- Inference & Predict: load model → generate state‐transition predictions
F. External Services & Tools
- Weights & Biases (wandb) for experiment tracking
- Slurm integration (emb/tools/slurm.py) for HPC launches
- Vector database (LanceDB) for embedding storage & k-NN queries
- Singularity container for reproducibility

##### Relationships & Data Flows
1. User issues `state emb ...` or `state tx ...` at shell
2. CLI parses subcommand → loads TOML/YAML config → selects pipeline
3. Pipeline loads raw data → passes through preprocessing components → yields in‐memory/anndata artifacts
4. Training pipelines instantiate appropriate model class → wrap in PyTorch Lightning Trainer → run training → checkpoint → (optionally) log to wandb
5. Inference pipelines load checkpoint → run forward passes → write out embeddings or perturbed expression predictions
6. Embeddings can be ingested into LanceDB; query pipeline reads embeddings → performs k-NN search → outputs .csv

##### Architectural Patterns & Principles
- CLI command‐group pattern (entrypoint + subcommands)
- Config‐driven orchestration (TOML + YAML)
- Modular “plugin” model registry (each model lives in its own module)
- Separation of concerns: data, models, training loop, evaluation/inference
- Heavy use of PyTorch Lightning for trainer abstraction

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI and Config Layer
    CLI["CLI Frontend\nstate"]:::blue
    click CLI "https://github.com/arcinstitute/state/blob/main/src/state/__main__.py"
    Config["Config Manager\n(CLI flags + TOML/YAML defaults)"]:::blue
    click Config "https://github.com/arcinstitute/state/blob/main/src/state/configs/config.yaml"

    CLI -->|"dispatch to subcommands"| Config

    %% Embedding Preprocess CLI
    EmbPreprocCLI["emb preprocess CLI"]:::green
    click EmbPreprocCLI "https://github.com/arcinstitute/state/blob/main/src/state/_cli/_emb/_preprocess.py"
    Config --> EmbPreprocCLI

    %% Transition Preprocess CLI
    TxPreprocTrainCLI["tx preprocess_train CLI"]:::green
    click TxPreprocTrainCLI "https://github.com/arcinstitute/state/blob/main/src/state/_cli/_tx/_preprocess_train.py"
    TxPreprocInferCLI["tx preprocess_infer CLI"]:::green
    click TxPreprocInferCLI "https://github.com/arcinstitute/state/blob/main/src/state/_cli/_tx/_preprocess_infer.py"
    Config --> TxPreprocTrainCLI
    Config --> TxPreprocInferCLI

    %% Data Layer
    RawH5AD1(["Raw h5ad files"]):::cylinder
    EmbPreprocCLI -->|"reads raw h5ad"| RawH5AD1
    EmbDataLoader["Embedding Data Loader"]:::green
    click EmbDataLoader "https://github.com/arcinstitute/state/blob/main/src/state/emb/data/loader.py"
    RawH5AD1 -->|"processed h5ad"| EmbDataLoader

    RawH5AD2(["Raw h5ad files"]):::cylinder
    TxPreprocTrainCLI -->|"reads raw h5ad"| RawH5AD2
    TxPreprocInferCLI -->|"reads raw h5ad"| RawH5AD2
    TxDataset["Transition Dataset"]:::green
    click TxDataset "https://github.com/arcinstitute/state/blob/main/src/state/tx/data/dataset/scgpt_perturbation_dataset.py"
    RawH5AD2 -->|"processed h5ad"| TxDataset

    %% Embedding Subsystem
    subgraph "Embedding Subsystem"
        direction TB
        EmbModel["Embedding Model\n(flash_transformer, decoder)"]:::orange
        click EmbModel "https://github.com/arcinstitute/state/blob/main/src/state/emb/nn/flash_transformer.py"
        EmbTrainer["Trainer\n(PyTorch Lightning)"]:::orange
        click EmbTrainer "https://github.com/arcinstitute/state/blob/main/src/state/emb/train/trainer.py"
        EmbCallbacks["Callbacks"]:::orange
        click EmbCallbacks "https://github.com/arcinstitute/state/blob/main/src/state/emb/train/callbacks.py"
        EmbCheckpoint(["Checkpoint (.ckpt)"]):::cylinder
        EmbInferCLI["emb transform CLI"]:::purple
        click EmbInferCLI "https://github.com/arcinstitute/state/blob/main/src/state/_cli/_emb/_transform.py"
        EmbInference["Inference Module"]:::purple
        click EmbInference "https://github.com/arcinstitute/state/blob/main/src/state/emb/inference.py"
        VectorDB["VectorDB Builder & Query"]:::purple
        click VectorDB "https://github.com/arcinstitute/state/blob/main/src/state/emb/vectordb.py"
    end

    EmbDataLoader --> EmbModel
    EmbModel --> EmbTrainer
    EmbTrainer --> EmbCallbacks
    EmbTrainer --> EmbCheckpoint
    EmbCheckpoint --> EmbInferCLI
    EmbInferCLI --> EmbInference
    EmbInference --> VectorDB
    click EmbPreprocCLI "https://github.com/arcinstitute/state/blob/main/src/state/_cli/_emb/_preprocess.py"

    %% Transition Subsystem
    subgraph "Transition Subsystem"
        direction TB
        TxModelZoo["Model Zoo\n(CPA, SCVI, SCGPT, decoder_only, ...)"]:::orange
        click TxModelZoo "https://github.com/arcinstitute/state/blob/main/src/state/tx/models/base.py"
        TxTrainer["Trainer\n(PyTorch Lightning)"]:::orange
        TxCallbacks["Callbacks"]:::orange
        click TxCallbacks "https://github.com/arcinstitute/state/blob/main/src/state/tx/callbacks/batch_speed_monitor.py"
        TxCheckpoint(["Checkpoint (.ckpt)"]):::cylinder
        TxInferCLI["tx infer CLI"]:::purple
        click TxInferCLI "https://github.com/arcinstitute/state/blob/main/src/state/_cli/_tx/_infer.py"
        TxPredictCLI["tx predict CLI"]:::purple
        click TxPredictCLI "https://github.com/arcinstitute/state/blob/main/src/state/_cli/_tx/_predict.py"
    end

    TxDataset --> TxModelZoo
    TxModelZoo --> TxTrainer
    TxTrainer --> TxCallbacks
    TxTrainer --> TxCheckpoint
    TxCheckpoint --> TxInferCLI
    TxCheckpoint --> TxPredictCLI

    %% External Services & Tools
    subgraph "External Services & Tools"
        direction TB
        WandB["Weights & Biases"]:::gray
        Slurm["Slurm HPC Launcher"]:::gray
        click Slurm "https://github.com/arcinstitute/state/blob/main/src/state/emb/tools/slurm.py"
        Singularity["Singularity Container"]:::gray
    end

    EmbTrainer --> WandB
    TxTrainer --> WandB
    EmbTrainer --> Slurm
    CLI --> Singularity

    %% Data Stores
    LanceDB(["LanceDB Store"]):::cylinder
    VectorDB --> LanceDB

    %% Styles
    classDef blue fill:#cfe2f3,stroke:#6d9eeb,color:#0b5394;
    classDef green fill:#d9ead3,stroke:#93c47d,color:#38761d;
    classDef orange fill:#fce5cd,stroke:#f6b26b,color:#b45f06;
    classDef purple fill:#e4def8,stroke:#8e7cc3,color:#674ea7;
    classDef gray fill:#eeeeee,stroke:#aaaaaa,color:#666666;
    classDef cylinder shape:cylinder;
```
