How lbinan/perturb-fish works
Project type and purpose
• This is a scientific data-processing and analysis pipeline for the Perturb-FISH publication.
• It handles three separate experimental datasets (THP1, astrocyte, tumor xenograft) plus an RNAscope module. Each dataset follows the same two major stages: image preprocessing → count-table generation, and downstream statistical analysis/visualization.
Main components
A. Data inputs
• Raw microscopy images (multiple rounds, modalities—MERFISH, perturbation channels, calcium imaging for astrocytes)
• Codebook definitions (mapping barcodes to perturbations)
B. Preprocessing modules (per dataset)
• THP1Preprocessing (MATLAB .m scripts)
• astrocyte_preprocessing (MATLAB .m scripts + registration routines)
• tumorpreprocessing (MATLAB .m scripts)
• RNAscope (MATLAB .m scripts for small-image generation and printing)
• dataexploration (MATLAB .m for threshold estimation)
C. Intermediate artefacts
• Filtered & corrected images
• Decoded barcodes (binary → perturbation IDs)
• Spot QC reports (intensities, standard deviations)
• Stitched cell mosaics (nuclei masks and transcript density masks)
• Global coordinate mappings
• Count tables (perturbations, RNA transcripts, calcium traces)
D. Analysis modules
• THP1analysis (MATLAB & Python scripts, .txt driver files)
• astrocyteanalysis (MATLAB .m scripts for cross-modality matching and effect inference)
• tumoranalysis (Python and Jupyter notebooks, MATLAB .m cluster analyses)
E. Final outputs
• Statistical summaries of perturbation effects
• Spatial-effect plots, correlation checks
• Heatmaps, clustering visualizations
Relationships and data flow
• Within preprocessing:
a. Image filtering & alignment (filter scripts) →
b. Threshold determination (manual or dataexploration) →
c. Decoding (decodeHW4* scripts) →
d. Spot QC →
e. Mosaic stitching & segmentation →
f. Coordinate translation (registration for astrocyte & tumor) →
g. Export count tables
• Analysis:
• Load count tables into respective analysis modules →
• Random splits & consistency checks →
• Effect size estimation (Python inference calls) →
• Cross-modality matching (astrocyte, tumor) →
• Plot generation (MATLAB and Jupyter)
Architectural patterns & design principles
• Modular pipeline: clear separation of preprocessing vs. analysis
• Dataset-specific sub-pipelines sharing common preprocessing steps (filter, decode, QC, segment)
• Functional decomposition: each script does a targeted task, enabling iterative refinement (e.g., threshold tuning)
• Polyglot tooling: MATLAB for image processing and initial analyses; Python/Jupyter for statistical inference and interactive visualization
Elements
• Processes (rectangles)
Detail without over-splitting
• But avoid drawing every single .m file: group logically (e.g., “Threshold & Decode” rather than “decodeHW4…m”).
• Dataset lane reveals its unique registration and analysis subflows.