# lbinan/perturb-fish architecture

> Architecture diagram of the GitHub repository lbinan/perturb-fish, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lbinan/perturb-fish
- Repository: https://github.com/lbinan/perturb-fish

## Overview

##### Project type and purpose
- This is a scientific data-processing and analysis pipeline for the Perturb-FISH publication.
- It handles three separate experimental datasets (THP1, astrocyte, tumor xenograft) plus an RNAscope module. Each dataset follows the same two major stages: image preprocessing → count-table generation, and downstream statistical analysis/visualization.

##### Main components
A. Data inputs
- Raw microscopy images (multiple rounds, modalities—MERFISH, perturbation channels, calcium imaging for astrocytes)
- Codebook definitions (mapping barcodes to perturbations)

B. Preprocessing modules (per dataset)
- THP1Preprocessing (MATLAB .m scripts)
- astrocyte_preprocessing (MATLAB .m scripts + registration routines)
- tumorpreprocessing (MATLAB .m scripts)
- RNAscope (MATLAB .m scripts for small-image generation and printing)
- dataexploration (MATLAB .m for threshold estimation)

C. Intermediate artefacts
- Filtered & corrected images
- Decoded barcodes (binary → perturbation IDs)
- Spot QC reports (intensities, standard deviations)
- Stitched cell mosaics (nuclei masks and transcript density masks)
- Global coordinate mappings
- Count tables (perturbations, RNA transcripts, calcium traces)

D. Analysis modules
- THP1analysis (MATLAB & Python scripts, .txt driver files)
- astrocyteanalysis (MATLAB .m scripts for cross-modality matching and effect inference)
- tumoranalysis (Python and Jupyter notebooks, MATLAB .m cluster analyses)

E. Final outputs
- Statistical summaries of perturbation effects
- Spatial-effect plots, correlation checks
- Heatmaps, clustering visualizations

##### Relationships and data flow
- Within preprocessing:
a. Image filtering & alignment (filter scripts) →
b. Threshold determination (manual or dataexploration) →
c. Decoding (decodeHW4* scripts) →
d. Spot QC →
e. Mosaic stitching & segmentation →
f. Coordinate translation (registration for astrocyte & tumor) →
g. Export count tables

- Analysis:
- Load count tables into respective analysis modules →
- Random splits & consistency checks →
- Effect size estimation (Python inference calls) →
- Cross-modality matching (astrocyte, tumor) →
- Plot generation (MATLAB and Jupyter)

##### Architectural patterns & design principles
- Modular pipeline: clear separation of preprocessing vs. analysis
- Dataset-specific sub-pipelines sharing common preprocessing steps (filter, decode, QC, segment)
- Functional decomposition: each script does a targeted task, enabling iterative refinement (e.g., threshold tuning)
- Polyglot tooling: MATLAB for image processing and initial analyses; Python/Jupyter for statistical inference and interactive visualization

##### Elements
- Processes (rectangles)

##### Detail without over-splitting
- But avoid drawing every single .m file: group logically (e.g., “Threshold & Decode” rather than “decodeHW4…m”).
- Dataset lane reveals its unique registration and analysis subflows.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% THP1 Lane
    subgraph "THP1 Dataset"
        direction TB
        THP1_Preproc["THP1 Preprocessing"]:::matlab
        THP1_Filter["Filter Images"]:::matlab
        THP1_Threshold["Threshold & Decode"]:::manual
        THP1_QC["Spot QC"]:::matlab
        THP1_Stitch["Stitch & Segment"]:::matlab
        THP1_Export["Export Count Tables"]:::data
        THP1_Analysis["THP1 Analysis"]:::python
        THP1_Output["Final Figures & Reports"]:::data
    end

    THP1_Preproc --> THP1_Filter
    THP1_Filter --> THP1_Threshold
    THP1_Threshold --> THP1_QC
    THP1_QC --> THP1_Stitch
    THP1_Stitch --> THP1_Export
    THP1_Export --> THP1_Analysis
    THP1_Analysis --> THP1_Output

    click THP1_Preproc "https://github.com/lbinan/perturb-fish/tree/main/thp1Preprocessing/"
    click THP1_Analysis "https://github.com/lbinan/perturb-fish/tree/main/THP1analysis/"

    %% Astrocyte Lane
    subgraph "Astrocyte Dataset"
        direction TB
        Astro_Preproc["Astrocyte Preprocessing"]:::matlab
        Astro_Filter["Filter Images"]:::matlab
        Astro_Threshold["Threshold & Decode"]:::manual
        Astro_QC["Spot QC"]:::matlab
        Astro_Stitch["Stitch & Segment"]:::matlab
        Astro_Reg["Registration"]:::registration
        Astro_Export["Export Count Tables"]:::data
        Astro_Analysis["Astrocyte Analysis"]:::python
        Astro_Output["Final Figures & Reports"]:::data
    end

    Astro_Preproc --> Astro_Filter
    Astro_Filter --> Astro_Threshold
    Astro_Threshold --> Astro_QC
    Astro_QC --> Astro_Stitch
    Astro_Stitch --> Astro_Reg
    Astro_Reg --> Astro_Export
    Astro_Export --> Astro_Analysis
    Astro_Analysis --> Astro_Output

    click Astro_Preproc "https://github.com/lbinan/perturb-fish/tree/main/astrocyte_preprocessing/"
    click Astro_Analysis "https://github.com/lbinan/perturb-fish/tree/main/astrocyteanalysis/"

    %% Tumor Lane
    subgraph "Tumor Dataset"
        direction TB
        Tumor_Preproc["Tumor Preprocessing"]:::matlab
        Tumor_Filter["Filter Images"]:::matlab
        Tumor_Threshold["Threshold & Decode"]:::manual
        Tumor_QC["Spot QC"]:::matlab
        Tumor_Stitch["Stitch & Segment"]:::matlab
        Tumor_Reg["Registration"]:::registration
        Tumor_Export["Export Count Tables"]:::data
        Tumor_Analysis["Tumor Analysis"]:::python
        Tumor_Output["Final Figures & Reports"]:::data
    end

    Tumor_Preproc --> Tumor_Filter
    Tumor_Filter --> Tumor_Threshold
    Tumor_Threshold --> Tumor_QC
    Tumor_QC --> Tumor_Stitch
    Tumor_Stitch --> Tumor_Reg
    Tumor_Reg --> Tumor_Export
    Tumor_Export --> Tumor_Analysis
    Tumor_Analysis --> Tumor_Output

    click Tumor_Preproc "https://github.com/lbinan/perturb-fish/tree/main/tumorpreprocessing/"
    click Tumor_Analysis "https://github.com/lbinan/perturb-fish/tree/main/tumoranalysis/"

    %% RNAscope Lane
    subgraph "RNAscope Module"
        direction TB
        RNAscope_Preproc["RNAscope Preprocessing"]:::matlab
        RNAscope_Filter["Filter & Print"]:::matlab
        RNAscope_QC["QC & Generate Images"]:::matlab
        RNAscope_Output["Final RNAscope Outputs"]:::data
    end

    RNAscope_Preproc --> RNAscope_Filter
    RNAscope_Filter --> RNAscope_QC
    RNAscope_QC --> RNAscope_Output

    click RNAscope_Preproc "https://github.com/lbinan/perturb-fish/tree/main/RNAscope/"

    %% Shared Threshold Exploration
    dataexplore["Threshold Estimation"]:::manual
    click dataexplore "https://github.com/lbinan/perturb-fish/tree/main/dataexploration/"

    %% Legend (for class definitions)
    classDef matlab fill:#AED6F1,stroke:#21618C,color:#1B2631;
    classDef python fill:#ABEBC6,stroke:#1D8348,color:#1B2631;
    classDef data fill:#F9E79F,stroke:#B9770E,color:#1B2631;
    classDef manual fill:#FDEBD0,stroke:#B03A2E,color:#1B2631,stroke-dasharray: 5 5;
    classDef registration fill:#D2B4DE,stroke:#7D3C98,color:#1B2631;
```
