How 10xgenomics/spaceranger works
Major Components
• Input and Preprocessing: Represent modules that take raw sequencing files (fastq), imaging data (brightfield/fluorescence images), and reference files. This includes components within “bin” and “mro” (workflow definitions) that launch preprocessing steps such as generating fastq files and preparing references.
• Pipeline Orchestration: Highlight the “mro” directory which contains a large number of workflow definition files (the .mro scripts) that represent different pipeline stages (aggregator, analyzer, spatial, targeted, tenkit). This acts as an orchestration layer that sequences and controls execution.
• Core Processing Libraries: Illustrate critical computation modules found in “lib”. Show the separation by language and purpose:
▪ Python components: Located under “lib/python”, especially the “cellranger” modules, which contain algorithms for analysis (PCA, clustering, antibody and spatial analysis, etc.).
▪ Go components: Used for specific functionalities such as “sitecheck” and demultiplexing routines, indicating performance or concurrency requirements.
▪ Rust components: Represented in “lib/rust”; these are performance-centric modules handling barcode processing, demultiplexing, alignment, and statistical calculations. They include submodules (cr_aggr, cr_lib, spatial, etc.) that power core mathematical or I/O routines.
• Reporting and Web Summary Generation: Include the “websummary” and related directories. These take processed data and convert it into human-readable summaries, dashboards, metrics, and image processing outputs.
• Third-party Dependencies: Don’t forget external integrations and build dependencies (STAR, BWA, Conda environments, etc.) which are defined under “third-party”. This layer is critical to show where external tools are used or patched.
• Configuration and Build Tools: Mention configurations and build scripts (like Cargo.toml files, .cargo configuration, and build scripts in “third-party”) that orchestrate the various languages and deployment artifacts.
Relationships and Data Flow
• That once input data is preprocessed, it goes to analysis stages (cell-calling, demultiplexing, clustering) which are implemented across the Python, Go, and Rust layers.
• Link processing modules with each other: for instance, the output of the “tenkit” submodules (which handle fastq generation and preflight checks) flows into analysis steps in “cellranger” and then further into the reporting modules.
• That the workflow orchestration (mro files) acts as the conductor that triggers these independent modules in sequence.
Architectural Patterns and Design Principles
• Pipeline/Modular Architecture: Emphasize that the design is modular, with each component dedicated to a particular step in the analysis. This modularity is clear from the strict separation of stages (aggregator, analyzer, spatial, targeted, etc.)
• Use clear labels for each major block: “Input & Preprocessing”, “Workflow Orchestration (MRO)”, “Core Analysis – Python (cellranger)”, “Core Analysis – Rust (cr_lib, spatial, etc.)”, “Core Analysis – Go (sitecheck)”, “Reporting & Web Summary”, “Third-party Tools”, and “Configuration/Build Infrastructure”.