How scverse/scvi-tools works
This project is scvi-tools – a Python library designed for probabilistic analysis of single-cell omics data. It is not a traditional full-stack application with separate frontend and backend services; rather, it is a modular, research-focused library built on top of PyTorch (and other frameworks like Pyro and PyTorch Lightning) that provides a collection of building blocks for dimensionality reduction, data integration, automated annotation, and more.
Main components
• Core Library (src/scvi): This is the heart of the package. It is organized into submodules that include:
– Data Handling: Components in src/scvi/data and its nested directories manage data input/output, preprocessing, built-in datasets, and data utility functions.
– Dataloaders: Located in src/scvi/dataloaders, these classes are responsible for batching, sampling, and splitting the data.
– Models and Base Classes: Found in src/scvi/model and src/scvi/model/base; these contain various model implementations (e.g., scVI, TotalVI, AutoZI) and the underlying classes that govern model architecture and training logic.
– Modules: In src/scvi/module and its base submodules, which implement neural network components, VAE infrastructure, and specialized model-specific functions.
– Training Infrastructure: Under src/scvi/train, these components incorporate training plans, logging, progress tracking, and callbacks essential for model training.
– Distributions & Statistics: In src/scvi/distributions, which implement probabilistic distributions (negative binomial, beta binomial, etc.) used in the models.
– Criticism and External Integrations: Modules such as src/scvi/criticism and src/scvi/external provide additional model criticism tools and integrations with approaches like cellassign, contrastivevi, and others.
– Hub: Under src/scvi/hub, which likely facilitates model sharing, metadata management, and possibly model templates.
Relationships and interactions
• Data flows into the system via the data modules and built-in datasets. The raw inputs are processed and then fed into dataloaders.
• Dataloaders interface with model training components to supply data in a suitable format (batching, splitting) during the training process.
• Model modules leverage core base classes and architecture mixins (embeddings, training mixins, differential functionalities) to build various probabilistic models.
• The training package orchestrates training routines by invoking loggers, callbacks, and metric computations, incorporating the distributions defined separately.
• External modules (located in src/scvi/external) extend the capabilities of core models by introducing alternative modeling strategies and specialized functionality.
• Overarching configuration and core settings (e.g., src/scvi/_settings.py and _constants.py) standardize the runtime behavior and default parameters across components.
Architectural patterns and design principles
• Modular and Layered Design: The codebase is segregated into distinct modules (data, models, training, distributions, etc.), which communicate through clear interfaces.
• Reusability and Composability: The base classes in model and module directories are designed to be extended, fostering rapid development of new probabilistic models.
• Abstraction and Separation of Concerns: Each directory (e.g., data, model, train) encapsulates specific functionality, making it easier to test, extend, and maintain.
• Integration with External Frameworks: By building on top of PyTorch, Pyro, and PyTorch Lightning, scvi-tools leverages established high-performance libraries and design patterns.