# morris-lab/celloracle architecture

> Architecture diagram of the GitHub repository morris-lab/celloracle, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/morris-lab/celloracle
- Repository: https://github.com/morris-lab/celloracle

## Overview

##### Project type and purpose
- CellOracle is an open-source Python library (a toolkit) for in silico gene perturbation analyses using single-cell omics data and Gene Regulatory Network (GRN) models. It provides a modular pipeline from raw data ingestion through network inference and simulation to downstream analysis and visualization.

##### File-structure analysis
- Top-level packaging and CI:
- setup.py, requirements.txt, Dockerfile, Makefile, GitHub Actions workflows (build_check.yml, build_weekly_check.yml)
- Core code (celloracle/):
- data_conversion: R/Python scripts to convert Seurat or R objects into matrix formats
- data: loading reference genomes, motif and TSS annotation, example Anndata files
- motif_analysis: scanning peaks to TF binding information, processing Cicero and BED files
- network: core classes and utilities for building GRN scaffolds and regression models
- network_analysis: R‐script wrappers and utilities for computing network scores and cartography
- trajectory: oracle core, Markov simulation, pseudotime, Sankey transitions
- oracle_utility & applications: higher-level workflows (pseudotime, perturbation simulation, systematic analysis)
- visualizations: plotting modules for Oracle objects and development modules
- utility: generic helpers (I/O, HDF5, logging, package version checks)
- Test suite under celloracle/tests covers data loading, motif analysis, network construction, tutorial pipelines.

##### README insights
- Confirms supported reference genomes, external dependencies (Python ≥3.7, Scanpy, pandas, R packages via script wrappers), and use cases (ATAC-seq, scRNA-seq, simulation, pseudotime).
- Points to extensive documentation and tutorials, demonstrating Jupyter notebook–driven user interaction.

a. Main components (layers and modules)
- Input/External Data: raw single-cell omics (h5ad, Seurat), TF motif databases, reference genome annotations, downloaded via utility/data_download_from_web or prepackaged in celloracle/data
- Data Conversion Layer: process_seurat_object.py, seurat_to_mtx.R
- Data Loading Layer: load_data.py, load_promoter_base_GRN.py, config.py
- Motif Analysis: motif_data.py, process_bed_file.py, process_cicero_data.py, tfinfo_core.py
- GRN Scaffold/Construction: net_core.py, regression_models.py, network_construction.py
- Network Analysis: goatools_wrapper.py, use_r_scripts.py, gene_analysis, cartography
- Simulation & Trajectory Engine: oracle_core.py, markov_simulation.py, pseudotime modules, sankey
- Application Workflows: applications/*.py (pseudotime, differentiation_flow, systematic_analysis_helper)
- Visualization: oracle_object_visualization.py, development_module_visualization.py
- Utilities: logging, HDF5 handling, package checker
- Packaging/Deployment: setup.py, Dockerfile, GitHub Actions CI/CD

b. Relationships and data flow
1. Raw input → Data Conversion → standardized matrices
2. Standardized matrices + motif/reference files → Data Loading → in-memory Anndata or Pandas structures
3. Loaded data → Motif Analysis → TF-by-target info (parquet, pickle)
4. Motif results + promoter GRN data → GRN Scaffold → regression models infer edge weights
5. Inferred network → Network Analysis (R wrappers) → network metrics, cartography
6. Inferred network + cell embeddings → Simulation/Trajectory Engine → transition probabilities, pseudotime
7. Simulation outputs → Application Workflows → aggregate metrics, systematic perturbation results
8. Application outputs → Visualization → figures, interactive plots
9. Utility modules support logging, caching, downloads at each stage
10. Tests validate correctness at each step

c. Architectural patterns and design principles
- Layered/pipeline architecture: clearly separated data ingestion, processing, core logic, analysis, and visualization
- Modular design: each subpackage encapsulates a cohesive responsibility
- Extensibility via R script wrappers for network analysis and via application helper modules
- Data formats standardized (Anndata, parquet, pickle) for interoperability

d. Key technologies and frameworks
- Python ecosystem: pandas, numpy, scanpy/anndata, scikit-learn
- R ecosystem: igraph, goatools, linkcomm, newtork packages invoked via subprocess
- File formats: HDF5 (.h5ad), parquet, BED, text, pickle
- CI/CD: GitHub Actions, Docker, PyPI packaging

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% Input/External Data Layer
  subgraph "Input/External Data"
    A1["External Data\n(celloracle/data)"]:::input
    A2["Data Download Utility\n(data_download_from_web.py)"]:::input
  end

  %% Data Conversion Layer
  subgraph "Data Conversion Layer"
    B1["process_seurat_object.py"]:::core
    B2["seurat_to_mtx.R"]:::core
  end

  %% Data Loading Layer
  subgraph "Data Loading Layer"
    C1["load_data.py"]:::core
    C2["load_promoter_base_GRN.py"]:::core
    C3["config.py"]:::core
  end

  %% Motif Analysis
  subgraph "Motif Analysis"
    D1["motif_data.py"]:::analysis
    D2["process_bed_file.py"]:::analysis
    D3["process_cicero_data.py"]:::analysis
    D4["tfinfo_core.py"]:::analysis
    D5["tss_annotation & tss_ref_data"]:::analysis
  end

  %% GRN Scaffold / Construction
  subgraph "GRN Scaffold / Construction"
    E1["net_core.py"]:::core
    E2["regression_models.py"]:::core
    E3["net_util.py"]:::core
  end

  %% Network Analysis
  subgraph "Network Analysis"
    F1["network_construction.py"]:::analysis
    F2["cartography.py"]:::analysis
    F3["gene_analysis.py"]:::analysis
    F4["use_r_scripts.py"]:::analysis
    F5["rscripts_for_network_analysis"]:::analysis
  end

  %% Simulation & Trajectory Engine
  subgraph "Simulation & Trajectory Engine"
    G1["oracle_core.py"]:::sim
    G2["markov_simulation.py"]:::sim
    G3["oracle_GRN.py"]:::sim
    G4["sankey.py"]:::sim
    G5["neighbors & oracle_utility"]:::sim
  end

  %% Application Workflows
  subgraph "Application Workflows"
    H1["pseudotime.py"]:::core
    H2["differentiation_flow.py"]:::core
    H3["systematic_analysis_helper.py"]:::core
    H4["development_module.py"]:::core
  end

  %% Visualization
  subgraph "Visualization"
    I1["oracle_object_visualization.py"]:::viz
    I2["development_module_visualization.py"]:::viz
    I3["config.py"]:::viz
  end

  %% Utilities
  subgraph "Utilities"
    J1["make_log.py"]:::util
    J2["hdf5_processing.py"]:::util
    J3["load_hdf5.py"]:::util
    J4["package_version_checker.py"]:::util
    J5["utility.py"]:::util
  end

  %% Packaging & CI/CD
  subgraph "Packaging & CI/CD"
    K1["setup.py"]:::pkg
    K2["Dockerfile"]:::pkg
    K3["build_check.yml"]:::pkg
    K4["build_weekly_check.yml"]:::pkg
    K5["requirements.txt"]:::pkg
  end

  %% Data Flow
  A1 -->|provides raw files| B1
  A1 -->|provides raw files| B2
  A2 -->|downloads data| A1

  B1 -->|mtx outputs| C1
  B2 -->|mtx outputs| C1

  C1 -->|Anndata/pandas| D1
  C2 -->|GRN scaffold base| E1
  C3 -->|config| C1

  D1 --> D2
  D1 --> D3
  D1 --> D4
  D5 --> D4
  D2 --> E1
  D3 --> E1
  D4 --> E1

  E1 --> E2
  E2 --> F1
  E3 --> E1

  F1 --> F2
  F1 --> F3
  F4 --> F1
  F5 --> F4
  F2 --> G1
  F3 --> G1

  G1 --> G2
  G1 --> G3
  G2 --> G4
  G5 --> G1

  G4 --> H1
  G3 --> H1
  H1 --> H2
  H2 --> H3
  H3 --> H4

  H4 --> I1
  H4 --> I2
  I3 --> I1
  I3 --> I2

  J1 --> B1
  J2 --> C1
  J3 --> C1
  J4 --> C1
  J5 --> all

  K1 --> all
  K2 --> all
  K3 --> all
  K4 --> all
  K5 --> all

  %% Click Events
  click A1 "https://github.com/morris-lab/celloracle/tree/master/celloracle/data"
  click A2 "https://github.com/morris-lab/celloracle/blob/master/celloracle/utility/data_download_from_web.py"
  click B1 "https://github.com/morris-lab/celloracle/blob/master/celloracle/data_conversion/process_seurat_object.py"
  click B2 "https://github.com/morris-lab/celloracle/blob/master/celloracle/data_conversion/seurat_to_mtx.R"
  click C1 "https://github.com/morris-lab/celloracle/blob/master/celloracle/data/load_data.py"
  click C2 "https://github.com/morris-lab/celloracle/blob/master/celloracle/data/load_promoter_base_GRN.py"
  click C3 "https://github.com/morris-lab/celloracle/blob/master/celloracle/data/config.py"
  click D1 "https://github.com/morris-lab/celloracle/blob/master/celloracle/motif_analysis/motif_data.py"
  click D2 "https://github.com/morris-lab/celloracle/blob/master/celloracle/motif_analysis/process_bed_file.py"
  click D3 "https://github.com/morris-lab/celloracle/blob/master/celloracle/motif_analysis/process_cicero_data.py"
  click D4 "https://github.com/morris-lab/celloracle/blob/master/celloracle/motif_analysis/tfinfo_core.py"
  click D5 "https://github.com/morris-lab/celloracle/blob/master/celloracle/motif_analysis/tss_annotation.py"
  click E1 "https://github.com/morris-lab/celloracle/blob/master/celloracle/network/net_core.py"
  click E2 "https://github.com/morris-lab/celloracle/blob/master/celloracle/network/regression_models.py"
  click E3 "https://github.com/morris-lab/celloracle/blob/master/celloracle/network/net_util.py"
  click F1 "https://github.com/morris-lab/celloracle/blob/master/celloracle/network_analysis/network_construction.py"
  click F2 "https://github.com/morris-lab/celloracle/blob/master/celloracle/network_analysis/cartography.py"
  click F3 "https://github.com/morris-lab/celloracle/blob/master/celloracle/network_analysis/gene_analysis.py"
  click F4 "https://github.com/morris-lab/celloracle/blob/master/celloracle/network_analysis/use_r_scripts.py"
  click F5 "https://github.com/morris-lab/celloracle/tree/master/celloracle/network_analysis/rscripts_for_network_analysis/"
  click G1 "https://github.com/morris-lab/celloracle/blob/master/celloracle/trajectory/oracle_core.py"
  click G2 "https://github.com/morris-lab/celloracle/blob/master/celloracle/trajectory/markov_simulation.py"
  click G3 "https://github.com/morris-lab/celloracle/blob/master/celloracle/trajectory/oracle_GRN.py"
  click G4 "https://github.com/morris-lab/celloracle/blob/master/celloracle/trajectory/sankey.py"
  click G5 "https://github.com/morris-lab/celloracle/blob/master/celloracle/trajectory/neighbors.py"
  click H1 "https://github.com/morris-lab/celloracle/blob/master/celloracle/applications/pseudotime.py"
  click H2 "https://github.com/morris-lab/celloracle/blob/master/celloracle/applications/differentiation_flow.py"
  click H3 "https://github.com/morris-lab/celloracle/blob/master/celloracle/applications/systematic_analysis_helper.py"
  click H4 "https://github.com/morris-lab/celloracle/blob/master/celloracle/applications/development_module.py"
  click I1 "https://github.com/morris-lab/celloracle/blob/master/celloracle/visualizations/oracle_object_visualization.py"
  click I2 "https://github.com/morris-lab/celloracle/blob/master/celloracle/visualizations/development_module_visualization.py"
  click I3 "https://github.com/morris-lab/celloracle/blob/master/celloracle/visualizations/config.py"
  click J1 "https://github.com/morris-lab/celloracle/blob/master/celloracle/utility/make_log.py"
  click J2 "https://github.com/morris-lab/celloracle/blob/master/celloracle/utility/hdf5_processing.py"
  click J3 "https://github.com/morris-lab/celloracle/blob/master/celloracle/utility/load_hdf5.py"
  click J4 "https://github.com/morris-lab/celloracle/blob/master/celloracle/utility/package_version_checker.py"
  click J5 "https://github.com/morris-lab/celloracle/blob/master/celloracle/utility/utility.py"
  click K1 "https://github.com/morris-lab/celloracle/blob/master/setup.py"
  click K2 "https://github.com/morris-lab/celloracle/tree/master/Dockerfile"
  click K3 "https://github.com/morris-lab/celloracle/blob/master/.github/workflows/build_check.yml"
  click K4 "https://github.com/morris-lab/celloracle/blob/master/.github/workflows/build_weekly_check.yml"
  click K5 "https://github.com/morris-lab/celloracle/blob/master/requirements.txt"

  %% Styles
  classDef input fill:#D6E9F8,stroke:#333
  classDef core fill:#DFF0D8,stroke:#333
  classDef analysis fill:#FCF8E3,stroke:#333
  classDef sim fill:#FAEBCC,stroke:#333
  classDef viz fill:#F2DEDE,stroke:#333
  classDef util fill:#EBEDF0,stroke:#333
  classDef pkg fill:#EEEEEE,stroke:#333
```
