# qiqi-lu/fluoresfm architecture

> Architecture diagram of the GitHub repository qiqi-lu/fluoresfm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/qiqi-lu/fluoresfm
- Repository: https://github.com/qiqi-lu/fluoresfm

## Overview

1. This repository is organized as a research pipeline around one core product: FluoResFM, a text-conditioned U-Net for fluorescence microscopy restoration. The pipeline spans data preparation, prompt generation, training, inference, evaluation, visualization, and downstream analysis.

2. The main model path is in `models/` and `utils/`. `models/` contains the restoration backbones and building blocks: U-Net variants, RCAN, DFCAN, SwinIR, CARE, DDN, autoencoder, PSF estimator, and text/image embedding modules. `utils/` holds reusable training and evaluation helpers such as data handling, losses, optimization, plotting, and analysis routines.

3. The central data flow starts from raw fluorescence images and paired ground truth images. Preprocessing scripts at the repository root (`0_*`) and under `datasets/` convert images into patches, clean background-only samples, normalize formats, add channel axes, convert between 2D/3D views, and store dataset metadata. The `dataset_*xlsx` files act as the key tabular source for dataset splits, structure labels, tasks, and microscope conditions.

4. Text conditioning is a distinct subsystem. Scripts like `1_0_text_generation.py`, `1_1_download_embedder.py`, `1_2_text_embedding.py`, and `1_4_prompt_tokenize.py` generate structured prompts from dataset metadata and embed them with BiomedCLIP-derived encoders. These text embeddings become the conditioning signal for the restoration model.

5. Training is split into two architectural variants: text-conditioned image-to-image training (`2_0_train_it2i.py`) and image-only baselines (`2_1_train_i2i.py`). The training scripts are paired with `3_*` test scripts for internal/external datasets, and finetuning is handled as a special case where only selected layers are updated, consistent with the README’s “fire marker” fine-tuning boundary.

6. Evaluation is a separate stage rather than being embedded in training. Scripts in the `4_*` range compute per-sample metrics, aggregate them across datasets, and summarize results by dataset, task, and resolution. Outputs are typically written back to spreadsheet-style artifacts for later analysis and figure generation.

7. Visualization and result inspection form another layer of the pipeline. The `5_*`, `7_*`, and `6_*` scripts generate box plots, radar plots, violin plots, scatter plots, image comparison panels, finetuning curves, mask overlays, and segmentation quality summaries. These scripts consume the metric tables produced by evaluation and turn them into publication-ready figures.

8. The repository also includes a downstream segmentation workflow. `6_0_segment.py` and related scripts use external segmentation systems such as Cellpose-SAM and Nellie to segment cells, nuclei, and intracellular organelles on restored or original images, then evaluate and visualize the masks. This makes FluoResFM a preprocessing component for morphology analysis, not just an end model.

9. A classification subsystem supports retrieval-based structure prediction. The `11_*` scripts construct a classification dataset, embed images, perform manual annotation, predict structure types, and visualize classification data/results. This subsystem exists to reduce prompt ambiguity before restoration and to support better text conditioning.

10. Several scripts focus on morphology-aware measurements rather than restoration quality. The `13_*` scripts compute structure-specific measurements such as CCP diameter, ER node statistics, MT network length, and text-prompt confusion analysis. These are downstream consumers of either restored images or segmentation outputs.

11. The `datasets/` directory contains data-reshaping utilities and format converters, including 2D/3D slicing, patch creation, projection, TIFF conversion, grayscale conversion, and downsampling. This folder is effectively the data engineering layer feeding both training and evaluation.

12. The `methods/` and `packages/` directories are vendored dependencies and legacy analysis toolchains, not the core FluoResFM pipeline. `packages/flash-attention-main` provides the performance-critical attention backend and its training examples, while `packages/nellie-main` packages Nellie’s segmentation and tracking stack. `methods/SIFNE` and `methods/MeshworkAnalyzer` contain MATLAB/IDL-based legacy morphology tools and are used when organelle/network analysis needs specialized algorithms.

13. The operational boundaries are clear: Excel metadata drives dataset construction, text embeddings condition the model, PyTorch/Triton/flash-attn execute training and inference on GPU, and separate evaluation scripts consume saved predictions to produce metrics and figures. This separation makes the repo more like a reproducible experimental platform than a single deployable application.

14. If you draw an architecture diagram, the most important nodes are: dataset metadata and image patches on the left; text generation/embedding as the conditioning branch; training/inference as the central model branch; evaluation and visualization as the reporting branch; and segmentation/classification/morphology analysis as downstream consumers of restored images. The vendored `flash-attention` and `nellie` packages should be shown as external-but-in-repo dependencies supporting the main pipeline.

Main entry points: [`models/unet_sd_ct.py`](https://github.com/qiqi-lu/fluoresfm/blob/main/models/unet_sd_ct.py) (FluoResFM), [`3_0_test_it2i.py`](https://github.com/qiqi-lu/fluoresfm/blob/main/3_0_test_it2i.py) (Inference), [`datasets/preprocessing.py`](https://github.com/qiqi-lu/fluoresfm/blob/main/datasets/preprocessing.py) (Preprocess), [`6_0_segment.py`](https://github.com/qiqi-lu/fluoresfm/blob/main/6_0_segment.py) (Segmentation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g_data["Data prep"]
  node_n_meta[("Dataset tables<br/>xlsx metadata")]
  node_n_prep["Preprocess<br/>data prep scripts<br/>[preprocessing.py]"]
  node_n_ds_utils["Data utils<br/>dataset helpers<br/>[utils_data.py]"]
end

subgraph group_g_text["Text conditioning"]
  node_n_textgen["Prompt build<br/>prompt synthesis"]
  node_n_embedder["Embedder<br/>biomedclip encoder"]
  node_n_textemb["Text embed<br/>conditioning embeddings"]
  node_n_tokenize["Tokenize<br/>prompt tokens"]
end

subgraph group_g_model["Core models"]
  node_n_unet["FluoResFM<br/>text-conditioned U-Net<br/>[unet_sd_ct.py]"]
  node_n_backbones["Backbones<br/>restoration models<br/>[rcan2d.py]"]
  node_n_embmods["Embed modules<br/>feature encoders"]
end

subgraph group_g_train["Train/test"]
  node_n_train_t["Train IT2I<br/>text-conditioned training<br/>[2_0_train_it2i.py]"]
  node_n_train_i["Train I2I<br/>baseline training<br/>[2_1_train_i2i.py]"]
  node_n_test["Inference<br/>test scripts<br/>[3_0_test_it2i.py]"]
end

subgraph group_g_eval["Evaluation"]
  node_n_eval["Metrics<br/>evaluation pipeline"]
end

subgraph group_g_viz["Visualization"]
  node_n_viz["Figures<br/>plots and panels"]
end

subgraph group_g_downstream["Downstream analysis"]
  node_n_segment["Segmentation<br/>mask workflow<br/>[6_0_segment.py]"]
  node_n_classify["Classification<br/>structure prediction"]
  node_n_morph["Morphology<br/>measurements"]
end

subgraph group_g_vendor["Vendored deps"]
  node_n_flash["flash-attn<br/>GPU attention backend"]
  node_n_nellie["Nellie<br/>segmentation package<br/>[networking.py]"]
end

node_n_meta -->|"drives"| node_n_prep
node_n_prep -->|"uses"| node_n_ds_utils
node_n_meta -->|"feeds"| node_n_textgen
node_n_textgen -->|"formats"| node_n_tokenize
node_n_tokenize -->|"conditions"| node_n_textemb
node_n_embedder -->|"backs"| node_n_textemb
node_n_textemb -->|"conditions"| node_n_unet
node_n_prep -->|"provides data"| node_n_unet
node_n_backbones -->|"baseline models"| node_n_train_i
node_n_unet -->|"trained by"| node_n_train_t
node_n_backbones -->|"compared in"| node_n_train_t
node_n_train_t -->|"exports"| node_n_test
node_n_train_i -->|"exports"| node_n_test
node_n_test -->|"summarizes"| node_n_eval
node_n_eval -->|"visualized by"| node_n_viz
node_n_test -->|"downstream input"| node_n_segment
node_n_segment -->|"uses"| node_n_nellie
node_n_segment -->|"enables"| node_n_morph
node_n_classify -.->|"reduces ambiguity"| node_n_textgen
node_n_classify -.->|"supports conditioning"| node_n_unet
node_n_morph -->|"reported in"| node_n_viz
node_n_flash -->|"accelerates"| node_n_unet

click node_n_meta "https://github.com/qiqi-lu/fluoresfm/blob/main/dataset_train_transformer-v2.xlsx"
click node_n_prep "https://github.com/qiqi-lu/fluoresfm/blob/main/datasets/preprocessing.py"
click node_n_ds_utils "https://github.com/qiqi-lu/fluoresfm/blob/main/datasets/utils_data.py"
click node_n_textgen "https://github.com/qiqi-lu/fluoresfm/blob/main/1_0_text_generation.py"
click node_n_embedder "https://github.com/qiqi-lu/fluoresfm/blob/main/1_1_download_embedder.py"
click node_n_textemb "https://github.com/qiqi-lu/fluoresfm/blob/main/1_2_text_embedding.py"
click node_n_tokenize "https://github.com/qiqi-lu/fluoresfm/blob/main/1_4_prompt_tokenize.py"
click node_n_unet "https://github.com/qiqi-lu/fluoresfm/blob/main/models/unet_sd_ct.py"
click node_n_backbones "https://github.com/qiqi-lu/fluoresfm/blob/main/models/rcan2d.py"
click node_n_embmods "https://github.com/qiqi-lu/fluoresfm/blob/main/models/biomedclip_embedder.py"
click node_n_train_t "https://github.com/qiqi-lu/fluoresfm/blob/main/2_0_train_it2i.py"
click node_n_train_i "https://github.com/qiqi-lu/fluoresfm/blob/main/2_1_train_i2i.py"
click node_n_test "https://github.com/qiqi-lu/fluoresfm/blob/main/3_0_test_it2i.py"
click node_n_eval "https://github.com/qiqi-lu/fluoresfm/blob/main/4_0_result_evaluate.py"
click node_n_viz "https://github.com/qiqi-lu/fluoresfm/blob/main/5_0_display_metrics_each_dataset.py"
click node_n_segment "https://github.com/qiqi-lu/fluoresfm/blob/main/6_0_segment.py"
click node_n_classify "https://github.com/qiqi-lu/fluoresfm/blob/main/11_0_classification_dataset_construct.py"
click node_n_morph "https://github.com/qiqi-lu/fluoresfm/blob/main/13_0_analysis_ccp_single_sample.py"
click node_n_flash "https://github.com/qiqi-lu/fluoresfm/blob/main/packages/flash-attention-main/flash_attn/flash_attn_interface.py"
click node_n_nellie "https://github.com/qiqi-lu/fluoresfm/blob/main/packages/nellie-main/nellie/segmentation/networking.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n_meta,node_n_prep,node_n_ds_utils,node_n_segment,node_n_classify,node_n_morph toneBlue
class node_n_textgen,node_n_embedder,node_n_textemb,node_n_tokenize,node_n_flash,node_n_nellie toneAmber
class node_n_unet,node_n_backbones,node_n_embmods toneMint
class node_n_train_t,node_n_train_i,node_n_test toneRose
class node_n_eval toneIndigo
class node_n_viz toneTeal
```

## Components

### Data prep

Raw fluorescence images, metadata tables, and preprocessing scripts that build model-ready datasets.

- [Dataset tables](https://github.com/qiqi-lu/fluoresfm/blob/main/dataset_train_transformer-v2.xlsx): xlsx metadata · Tabular dataset splits, structure labels, tasks, and microscope conditions. · `dataset_train_transformer-v2.xlsx`
- [Preprocess](https://github.com/qiqi-lu/fluoresfm/blob/main/datasets/preprocessing.py): data prep scripts · Cleaning, patching, normalization, channel-axis conversion, projections, and format transforms. · `datasets/preprocessing.py`
- [Data utils](https://github.com/qiqi-lu/fluoresfm/blob/main/datasets/utils_data.py): dataset helpers · Reusable dataset loading and reshaping utilities for preprocessing and training. · `datasets/utils_data.py`

### Text conditioning

Prompt creation, embedding, and tokenization for text-conditioned restoration.

- [Prompt build](https://github.com/qiqi-lu/fluoresfm/blob/main/1_0_text_generation.py): prompt synthesis · Generates structured text prompts from dataset metadata and task context. · `1_0_text_generation.py`
- [Embedder](https://github.com/qiqi-lu/fluoresfm/blob/main/1_1_download_embedder.py): biomedclip encoder · Downloads and prepares the text/image embedding backbone used for conditioning. · `1_1_download_embedder.py`
- [Text embed](https://github.com/qiqi-lu/fluoresfm/blob/main/1_2_text_embedding.py): conditioning embeddings · Turns prompts into vectors for model conditioning. · `1_2_text_embedding.py`
- [Tokenize](https://github.com/qiqi-lu/fluoresfm/blob/main/1_4_prompt_tokenize.py): prompt tokens · Tokenization and prompt formatting for the conditioning branch. · `1_4_prompt_tokenize.py`

### Core models

Restoration backbones and embedding modules used by training and inference.

- [FluoResFM](https://github.com/qiqi-lu/fluoresfm/blob/main/models/unet_sd_ct.py): text-conditioned U-Net · Main restoration model for fluorescence microscopy image-to-image restoration. · `models/unet_sd_ct.py`
- [Backbones](https://github.com/qiqi-lu/fluoresfm/blob/main/models/rcan2d.py): restoration models · Alternative restoration architectures such as U-Net variants, RCAN, DFCAN, SwinIR, CARE, DDN, and autoencoders. · `models/rcan2d.py`
- [Embed modules](https://github.com/qiqi-lu/fluoresfm/blob/main/models/biomedclip_embedder.py): feature encoders · Image and text embedding modules, including BiomedCLIP-derived components. · `models/biomedclip_embedder.py`

### Train/test

Training variants, inference scripts, and fine-tuning flows.

- [Train IT2I](https://github.com/qiqi-lu/fluoresfm/blob/main/2_0_train_it2i.py): text-conditioned training · Training loop for text-conditioned image restoration. · `2_0_train_it2i.py`
- [Train I2I](https://github.com/qiqi-lu/fluoresfm/blob/main/2_1_train_i2i.py): baseline training · Image-only restoration training for non-text baselines. · `2_1_train_i2i.py`
- [Inference](https://github.com/qiqi-lu/fluoresfm/blob/main/3_0_test_it2i.py): test scripts · Runs internal and external dataset inference, including fine-tuning evaluation paths. · `3_0_test_it2i.py`

### Evaluation

Metric aggregation and dataset-level analysis outputs.

- [Metrics](https://github.com/qiqi-lu/fluoresfm/blob/main/4_0_result_evaluate.py): evaluation pipeline · Computes per-sample and aggregated metrics across datasets, tasks, and resolutions. · `4_0_result_evaluate.py`

### Visualization

Figures, panels, and result inspection scripts.

- [Figures](https://github.com/qiqi-lu/fluoresfm/blob/main/5_0_display_metrics_each_dataset.py): plots and panels · Box plots, radar plots, violin plots, scatters, image comparisons, and finetuning curves. · `5_0_display_metrics_each_dataset.py`

### Downstream analysis

Segmentation, classification, and morphology-aware measurement workflows.

- [Segmentation](https://github.com/qiqi-lu/fluoresfm/blob/main/6_0_segment.py): mask workflow · Segments restored or original images and evaluates mask quality using external systems. · `6_0_segment.py`
- [Classification](https://github.com/qiqi-lu/fluoresfm/blob/main/11_0_classification_dataset_construct.py): structure prediction · Constructs image classification data, annotates, predicts structure types, and visualizes results. · `11_0_classification_dataset_construct.py`
- [Morphology](https://github.com/qiqi-lu/fluoresfm/blob/main/13_0_analysis_ccp_single_sample.py): measurements · Computes structure-specific morphology statistics and prompt confusion analyses. · `13_0_analysis_ccp_single_sample.py`

### Vendored deps

In-repo external packages supporting attention and segmentation.

- [flash-attn](https://github.com/qiqi-lu/fluoresfm/blob/main/packages/flash-attention-main/flash_attn/flash_attn_interface.py): GPU attention backend · Vendored high-performance attention kernels used by the model stack. · `packages/flash-attention-main/flash_attn/flash_attn_interface.py`
- [Nellie](https://github.com/qiqi-lu/fluoresfm/blob/main/packages/nellie-main/nellie/segmentation/networking.py): segmentation package · Vendored segmentation, tracking, and napari tooling used downstream. · `packages/nellie-main/nellie/segmentation/networking.py`

## Connections

- Dataset tables → Preprocess: drives · Metadata guides cleaning, patch creation, and dataset reshaping.
- Preprocess → Data utils: uses · Preprocessing relies on shared dataset helper routines.
- Dataset tables → Prompt build: feeds · Dataset tables are converted into structured prompts.
- Prompt build → Tokenize: formats · Generated prompts are normalized for tokenization.
- Tokenize → Text embed: conditions · Tokenized prompts are embedded into conditioning vectors.
- Embedder → Text embed: backs · The downloaded encoder provides the embedding backbone.
- Text embed → FluoResFM: conditions · Text embeddings guide the restoration model.
- Preprocess → FluoResFM: provides data · Prepared image patches become model inputs.
- Backbones → Train I2I: baseline models · Image-only training uses non-text restoration backbones.
- FluoResFM → Train IT2I: trained by · FluoResFM is optimized in the text-conditioned training loop.
- Backbones → Train IT2I: compared in · Alternative models serve as comparative restoration baselines.
- Train IT2I → Inference: exports · Trained checkpoints are evaluated through inference scripts.
- Train I2I → Inference: exports · Baseline checkpoints flow into the same test stage.
- Inference → Metrics: summarizes · Saved predictions are converted into metrics tables.
- Metrics → Figures: visualized by · Aggregated metrics power publication-style plots and panels.
- Inference → Segmentation: downstream input · Restored or original images are passed to segmentation workflows.
- Segmentation → Nellie: uses · Nellie provides segmentation and tracking capabilities.
- Segmentation → Morphology: enables · Masks and restored images feed morphology measurements.
- Classification → Prompt build: reduces ambiguity · Classification outputs support prompt selection and structure prediction.
- Classification → FluoResFM: supports conditioning · Predicted structure types help inform restoration prompts.
- Morphology → Figures: reported in · Morphology statistics are turned into figures and summaries.
- flash-attn → FluoResFM: accelerates · FlashAttention supports the central model’s GPU execution path.
