How arcinstitute/state works
Project Type and Purpose
• This is an open-source, command-line driven machine-learning toolkit (“State”) for:
a. Learning cell-state embeddings (the “emb” subcommand)
b. Training and running state-transition predictive models (the “tx” subcommand)
• Users configure experiments via TOML files; the system loads single-cell datasets in h5ad format, preprocesses them, trains PyTorch Lightning models, and runs inference or builds/query vector databases.
File‐Structure Overview
• Top level:
• README, LICENSE, singularity.def, scripts, examples, assets
• pyproject.toml, .github (CI), .gitmodules (vector DB submodule)
• src/state:
• __main__.py: entry point for state CLI
• _cli/: command‐group definitions for emb and tx (fit, transform, query, preprocess, train, infer, predict)
• configs/: YAML templates for data, models, training, wandb
• emb/: embedding pipeline (loader.py, inference.py, finetune_decoder.py, train/trainer.py, vectordb.py)
• tx/: transition pipeline (data preprocessing, dataset classes, many model modules, PyTorch Lightning tasks)
Key Components
A. CLI Frontend
• Single binary “state” → dispatches to two command groups: emb and tx
B. Configuration Layer
• Loads CLI flags + TOML configs → merges with YAML defaults
C. Data Layer
• Reads raw h5ad files (anndata) → preprocess (normalize, log1p, HVG selection) → outputs processed h5ad or control templates
D. Embedding Subsystem (emb)
• Data Loader → Embedding Model (flash_transformer, decoder) → Trainer (PyTorch Lightning) → Checkpoints
• Inference/Transform: load checkpoint → generate embeddings → optional vector database operations
• VectorDB Builder & Query: uses LanceDB backend via vectordb.py
E. Transition Subsystem (tx)
• Preprocessing commands (preprocess_train, preprocess_infer)
• Dataset classes (scgpt_perturbation_dataset.py)
• Model Zoo: CPA, SCVI, SCGPT, decoder_only, etc. (each in its own module)
• LightningModel wrappers and Trainer → Checkpoints
• Inference & Predict: load model → generate state‐transition predictions
F. External Services & Tools
• Weights & Biases (wandb) for experiment tracking
• Slurm integration (emb/tools/slurm.py) for HPC launches
• Vector database (LanceDB) for embedding storage & k-NN queries
• Singularity container for reproducibility
Relationships & Data Flows
1. User issues state emb ... or state tx ... at shell
2. CLI parses subcommand → loads TOML/YAML config → selects pipeline
3. Pipeline loads raw data → passes through preprocessing components → yields in‐memory/anndata artifacts
4. Training pipelines instantiate appropriate model class → wrap in PyTorch Lightning Trainer → run training → checkpoint → (optionally) log to wandb
5. Inference pipelines load checkpoint → run forward passes → write out embeddings or perturbed expression predictions
6. Embeddings can be ingested into LanceDB; query pipeline reads embeddings → performs k-NN search → outputs .csv
Architectural Patterns & Principles
• CLI command‐group pattern (entrypoint + subcommands)
• Config‐driven orchestration (TOML + YAML)
• Modular “plugin” model registry (each model lives in its own module)
• Separation of concerns: data, models, training loop, evaluation/inference
• Heavy use of PyTorch Lightning for trainer abstraction