# theislab/cellflow architecture

> Architecture diagram of the GitHub repository theislab/cellflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/theislab/cellflow
- Repository: https://github.com/theislab/cellflow

## Overview

##### Project Type and Purpose
- This is a Python library (cellflow-tools) designed to predict phenotypic responses under various perturbations using flow matching.
- The tool’s applications include drug response modeling, genetic perturbation analysis, developmental studies, and more.
- The README and file structure indicate that this is not a full-blown web application but rather a computational framework with clear modular components for data processing, model training, and analysis.

##### File Structure for Architectural Components
- data: Handles data ingestion, dataloader utilities, and management.
- preprocessing: Implements gene embedding, PCA, weighted k-nearest neighbors, and other techniques.
- model & networks: Contain core modeling logic (e.g., _cellflow.py) and neural network components (like encoders and velocity fields).
- solvers: Implements methods such as OT flow matching and algorithms (e.g., _genot, _otfm).
- training: Manages the training pipelines, callbacks, and related utilities.
- metrics: Provides evaluation metrics for model performance.
- plotting: Contains code for visualizations to help interpret model predictions.
- external: Integrates with external tools (e.g., scVI) to extend capabilities.
- utils: Common helpers used throughout the framework.
- Other directories (docs, tests, configuration files, GitHub workflows) support documentation, testing, CI/CD, and the project’s development lifecycle.

a. Main Components:
- Input/Data Layer: Represent modules for data handling (src/cellflow/data and datasets integration).
- Preprocessing Module: Highlight the preprocessing routines (_gene_emb, _pca, _preprocessing, _wknn).
- Modeling and Network Layers: Include the “model” and “networks” components (with custom flow matching and encoder/set encoder functions) as the core computational engine.
- External Integrations: Show the “external” module that connects to tools like scVI.
- Training and Solver Modules: Encapsulate the “solvers” and “training” parts—this is where the learning routine, callbacks, and optimization live.
- Evaluation and Visualization: Represent “metrics” for evaluation and “plotting” for generating results.
- Utility and Support Services: Include the “utils” module that supports operations across the framework.

b. Relationships and Interactions:
- Data flows from input (e.g., raw data ingestion) to the preprocessing layer.
- Preprocessed data then feeds into the modeling and network layers.
- The model’s training loop (comprising training callbacks and solvers) refines parameters based on the data.
- Results and predictions then flow into evaluation (metrics) and visualization (plotting) modules.
- Integration points where external modules (scVI) extend the base functionality.

c. Architectural Patterns and Design Principles:
- The overall design emphasizes modularity—each component (data, preprocessing, modeling, etc.) is encapsulated in its own submodule.
- The pipeline suggests a sequential and well-partitioned workflow, ensuring reusability and easy integration of additional tools or extensions.
- Clear separation of concerns is observed, which makes the system maintainable and scalable.
- Highlighting dependency injection or utility modules that serve cross-cutting concerns (logging, constant definitions, type hints).

d. Relevant Technologies, Frameworks, and Libraries:
- Python is the core language.
- JAX is suggested (installation notes in the README) for high-performance numerical computing, indicating modern machine learning techniques.
- CI/CD tools (GitHub Actions, Codecov, pre-commit) although not part of runtime, they support the overall architecture and development pipelines.
- Documentation is generated using Sphinx—hinted at by the “docs” folder and extensions.

##### Specific Elements
- Clearly labeled nodes for: Input/Data, Preprocessing, Modeling/Networks, Training/Optimization (including Solvers), Evaluation/Metrics, Visualization/Plotting, and External/Integrations.
– Core computational modules (e.g., model, networks, training)
– Data handling and preprocessing stages
– External integrations (e.g., scVI)
– Supporting utilities (e.g., logging, configuration)

##### Final Advice
- Be very detailed: capture every aspect of the architecture by splitting the project into as many clearly defined components as possible.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Input/Data Layer"
        Input["Input & Data"]:::datalayer
        Datasets["Datasets Integration"]:::datalayer
    end

    subgraph "Preprocessing Module"
        Prep["Preprocessing"]:::preprocessing
    end

    subgraph "Modeling & Network Layers"
        Model["Modeling Core"]:::modeling
        Networks["Neural Network Components"]:::modeling
    end

    subgraph "Training & Solvers"
        Training["Training Pipeline"]:::training
        Solvers["Solvers (Flow Matching)"]:::training
    end

    subgraph "Evaluation & Visualization"
        Eval["Evaluation Metrics"]:::evaluation
        Plot["Visualization / Plotting"]:::evaluation
    end

    subgraph "External Integrations"
        External["External Tools (scVI, etc.)"]:::external
    end

    subgraph "Utility & Support"
        Utils["Utility Functions"]:::utility
        Logs["Logging"]:::utility
        Constants["Constants"]:::utility
    end

    Input -->|"ingests"| Prep
    Datasets -->|"provides"| Prep
    Prep -->|"feeds"| Model
    Prep -->|"feeds"| Networks
    Model -->|"triggers"| Training
    Networks -->|"triggers"| Training
    Training -->|"outputs"| Eval
    Training -->|"outputs"| Plot
    Solvers -->|"optimizes"| Training
    External -->|"enhances"| Model
    External -->|"enhances"| Networks
    Utils ---|"supports"| Input
    Utils ---|"supports"| Prep
    Utils ---|"supports"| Model
    Utils ---|"supports"| Training
    Logs ---|"supplies"| Model
    Logs ---|"supplies"| Training

    click Input "https://github.com/theislab/cellflow/tree/main/src/cellflow/data"
    click Datasets "https://github.com/theislab/cellflow/blob/main/src/cellflow/datasets.py"
    click Prep "https://github.com/theislab/cellflow/tree/main/src/cellflow/preprocessing"
    click Model "https://github.com/theislab/cellflow/tree/main/src/cellflow/model"
    click Networks "https://github.com/theislab/cellflow/tree/main/src/cellflow/networks"
    click External "https://github.com/theislab/cellflow/tree/main/src/cellflow/external"
    click Training "https://github.com/theislab/cellflow/tree/main/src/cellflow/training"
    click Solvers "https://github.com/theislab/cellflow/tree/main/src/cellflow/solvers"
    click Eval "https://github.com/theislab/cellflow/tree/main/src/cellflow/metrics"
    click Plot "https://github.com/theislab/cellflow/tree/main/src/cellflow/plotting"
    click Utils "https://github.com/theislab/cellflow/blob/main/src/cellflow/utils.py"
    click Logs "https://github.com/theislab/cellflow/blob/main/src/cellflow/_logging.py"
    click Constants "https://github.com/theislab/cellflow/blob/main/src/cellflow/_constants.py"

    classDef datalayer fill:#add8e6,stroke:#000,stroke-width:1px
    classDef preprocessing fill:#ffcc80,stroke:#000,stroke-width:1px
    classDef modeling fill:#98fb98,stroke:#000,stroke-width:1px
    classDef training fill:#ffa07a,stroke:#000,stroke-width:1px
    classDef evaluation fill:#dda0dd,stroke:#000,stroke-width:1px
    classDef external fill:#ffffe0,stroke:#000,stroke-width:1px
    classDef utility fill:#d3d3d3,stroke:#000,stroke-width:1px
```
