How qiqi-lu/fluoresfm works
1. This repository is organized as a research pipeline around one core product: FluoResFM, a text-conditioned U-Net for fluorescence microscopy restoration. The pipeline spans data preparation, prompt generation, training, inference, evaluation, visualization, and downstream analysis.
2. The main model path is in models/ and utils/. models/ contains the restoration backbones and building blocks: U-Net variants, RCAN, DFCAN, SwinIR, CARE, DDN, autoencoder, PSF estimator, and text/image embedding modules. utils/ holds reusable training and evaluation helpers such as data handling, losses, optimization, plotting, and analysis routines.
3. The central data flow starts from raw fluorescence images and paired ground truth images. Preprocessing scripts at the repository root (0_*) and under datasets/ convert images into patches, clean background-only samples, normalize formats, add channel axes, convert between 2D/3D views, and store dataset metadata. The dataset_*xlsx files act as the key tabular source for dataset splits, structure labels, tasks, and microscope conditions.
4. Text conditioning is a distinct subsystem. Scripts like 1_0_text_generation.py, 1_1_download_embedder.py, 1_2_text_embedding.py, and 1_4_prompt_tokenize.py generate structured prompts from dataset metadata and embed them with BiomedCLIP-derived encoders. These text embeddings become the conditioning signal for the restoration model.
5. Training is split into two architectural variants: text-conditioned image-to-image training (2_0_train_it2i.py) and image-only baselines (2_1_train_i2i.py). The training scripts are paired with 3_* test scripts for internal/external datasets, and finetuning is handled as a special case where only selected layers are updated, consistent with the README’s “fire marker” fine-tuning boundary.
6. Evaluation is a separate stage rather than being embedded in training. Scripts in the 4_* range compute per-sample metrics, aggregate them across datasets, and summarize results by dataset, task, and resolution. Outputs are typically written back to spreadsheet-style artifacts for later analysis and figure generation.
7. Visualization and result inspection form another layer of the pipeline. The 5_*, 7_*, and 6_* scripts generate box plots, radar plots, violin plots, scatter plots, image comparison panels, finetuning curves, mask overlays, and segmentation quality summaries. These scripts consume the metric tables produced by evaluation and turn them into publication-ready figures.
8. The repository also includes a downstream segmentation workflow. 6_0_segment.py and related scripts use external segmentation systems such as Cellpose-SAM and Nellie to segment cells, nuclei, and intracellular organelles on restored or original images, then evaluate and visualize the masks. This makes FluoResFM a preprocessing component for morphology analysis, not just an end model.
9. A classification subsystem supports retrieval-based structure prediction. The 11_* scripts construct a classification dataset, embed images, perform manual annotation, predict structure types, and visualize classification data/results. This subsystem exists to reduce prompt ambiguity before restoration and to support better text conditioning.
10. Several scripts focus on morphology-aware measurements rather than restoration quality. The 13_* scripts compute structure-specific measurements such as CCP diameter, ER node statistics, MT network length, and text-prompt confusion analysis. These are downstream consumers of either restored images or segmentation outputs.
11. The datasets/ directory contains data-reshaping utilities and format converters, including 2D/3D slicing, patch creation, projection, TIFF conversion, grayscale conversion, and downsampling. This folder is effectively the data engineering layer feeding both training and evaluation.
12. The methods/ and packages/ directories are vendored dependencies and legacy analysis toolchains, not the core FluoResFM pipeline. packages/flash-attention-main provides the performance-critical attention backend and its training examples, while packages/nellie-main packages Nellie’s segmentation and tracking stack. methods/SIFNE and methods/MeshworkAnalyzer contain MATLAB/IDL-based legacy morphology tools and are used when organelle/network analysis needs specialized algorithms.
13. The operational boundaries are clear: Excel metadata drives dataset construction, text embeddings condition the model, PyTorch/Triton/flash-attn execute training and inference on GPU, and separate evaluation scripts consume saved predictions to produce metrics and figures. This separation makes the repo more like a reproducible experimental platform than a single deployable application.
14. If you draw an architecture diagram, the most important nodes are: dataset metadata and image patches on the left; text generation/embedding as the conditioning branch; training/inference as the central model branch; evaluation and visualization as the reporting branch; and segmentation/classification/morphology analysis as downstream consumers of restored images. The vendored flash-attention and nellie packages should be shown as external-but-in-repo dependencies supporting the main pipeline.
Main entry points: unet_sd_ct.py FluoResFM · 3_0_test_it2i.py Inference · preprocessing.py Preprocess · 6_0_segment.py Segmentation
Components 20
Data prep
Raw fluorescence images, metadata tables, and preprocessing scripts that build model-ready datasets.
- Dataset tables · Tabular dataset splits, structure labels, tasks, and microscope conditions.
dataset_train_transformer-v2.xlsx - Preprocess · Cleaning, patching, normalization, channel-axis conversion, projections, and format transforms.
datasets/preprocessing.py - Data utils · Reusable dataset loading and reshaping utilities for preprocessing and training.
datasets/utils_data.py
Text conditioning
Prompt creation, embedding, and tokenization for text-conditioned restoration.
- Prompt build · Generates structured text prompts from dataset metadata and task context.
1_0_text_generation.py - Embedder · Downloads and prepares the text/image embedding backbone used for conditioning.
1_1_download_embedder.py - Text embed · Turns prompts into vectors for model conditioning.
1_2_text_embedding.py - Tokenize · Tokenization and prompt formatting for the conditioning branch.
1_4_prompt_tokenize.py
Core models
Restoration backbones and embedding modules used by training and inference.
- FluoResFM · Main restoration model for fluorescence microscopy image-to-image restoration.
models/unet_sd_ct.py - Backbones · Alternative restoration architectures such as U-Net variants, RCAN, DFCAN, SwinIR, CARE, DDN, and autoencoders.
models/rcan2d.py - Embed modules · Image and text embedding modules, including BiomedCLIP-derived components.
models/biomedclip_embedder.py
Train/test
Training variants, inference scripts, and fine-tuning flows.
- Train IT2I · Training loop for text-conditioned image restoration.
2_0_train_it2i.py - Train I2I · Image-only restoration training for non-text baselines.
2_1_train_i2i.py - Inference · Runs internal and external dataset inference, including fine-tuning evaluation paths.
3_0_test_it2i.py
Evaluation
Metric aggregation and dataset-level analysis outputs.
- Metrics · Computes per-sample and aggregated metrics across datasets, tasks, and resolutions.
4_0_result_evaluate.py
Visualization
Figures, panels, and result inspection scripts.
- Figures · Box plots, radar plots, violin plots, scatters, image comparisons, and finetuning curves.
5_0_display_metrics_each_dataset.py
Downstream analysis
Segmentation, classification, and morphology-aware measurement workflows.
- Segmentation · Segments restored or original images and evaluates mask quality using external systems.
6_0_segment.py - Classification · Constructs image classification data, annotates, predicts structure types, and visualizes results.
11_0_classification_dataset_construct.py - Morphology · Computes structure-specific morphology statistics and prompt confusion analyses.
13_0_analysis_ccp_single_sample.py
Vendored deps
In-repo external packages supporting attention and segmentation.
- flash-attn · Vendored high-performance attention kernels used by the model stack.
packages/flash-attention-main/flash_attn/flash_attn_interface.py - Nellie · Vendored segmentation, tracking, and napari tooling used downstream.
packages/nellie-main/nellie/segmentation/networking.py
Connections 22
- Dataset tables → Preprocess: drives · Metadata guides cleaning, patch creation, and dataset reshaping.
- Preprocess → Data utils: uses · Preprocessing relies on shared dataset helper routines.
- Dataset tables → Prompt build: feeds · Dataset tables are converted into structured prompts.
- Prompt build → Tokenize: formats · Generated prompts are normalized for tokenization.
- Tokenize → Text embed: conditions · Tokenized prompts are embedded into conditioning vectors.
- Embedder → Text embed: backs · The downloaded encoder provides the embedding backbone.
- Text embed → FluoResFM: conditions · Text embeddings guide the restoration model.
- Preprocess → FluoResFM: provides data · Prepared image patches become model inputs.
- Backbones → Train I2I: baseline models · Image-only training uses non-text restoration backbones.
- FluoResFM → Train IT2I: trained by · FluoResFM is optimized in the text-conditioned training loop.
- Backbones → Train IT2I: compared in · Alternative models serve as comparative restoration baselines.
- Train IT2I → Inference: exports · Trained checkpoints are evaluated through inference scripts.
- Train I2I → Inference: exports · Baseline checkpoints flow into the same test stage.
- Inference → Metrics: summarizes · Saved predictions are converted into metrics tables.
- Metrics → Figures: visualized by · Aggregated metrics power publication-style plots and panels.
- Inference → Segmentation: downstream input · Restored or original images are passed to segmentation workflows.
- Segmentation → Nellie: uses · Nellie provides segmentation and tracking capabilities.
- Segmentation → Morphology: enables · Masks and restored images feed morphology measurements.
- Classification → Prompt build: reduces ambiguity · Classification outputs support prompt selection and structure prediction.
- Classification → FluoResFM: supports conditioning · Predicted structure types help inform restoration prompts.
- Morphology → Figures: reported in · Morphology statistics are turned into figures and summaries.
- flash-attn → FluoResFM: accelerates · FlashAttention supports the central model’s GPU execution path.