How hkxiao/zs-cosod works
Project type and purpose
• This is a research‐oriented Python deep‐learning pipeline for zero‐shot Co‐salient Object Detection (CoSOD). It is not a web or full‐stack app but a modular ML framework that leverages multiple open‐source models (Stable Diffusion, DINOv2, SAM, TSDN) to perform a series of data‐processing and inference steps without any training.
• The README describes two core algorithmic components: the Group Prompt Generation (GPG) module and the Co‐Saliency Map Generation (CMP) module, plus feature‐extraction and evaluation steps.
File structure
• Top‐level modules:
• sd-dino/: Stable Diffusion + DINOv2‐based feature extractor
• A2S-v2/: TSDN unsupervised saliency model (config, encoder, method factories)
• segment_anything/: SAM model and utilities
• cosal.py, main.py, evaluate.py, etc.: orchestration scripts for full CoSOD pipeline
• Shell scripts in root and subdirectories (extract_feat.sh, inference_sod.sh, inference_cosod.sh) orchestrate the flow.
• Utility modules (util.py, data.py, framework_factory.py) provide shared functions and factory patterns for model instantiation.
• No web server, database, or frontend; it’s a batch inference and evaluation pipeline.
Key insights from README
• Workflow:
1. Feature extraction via sd-dino (Stable Diffusion + DINOv2)
2. Saliency map generation via A2S-v2 (Unsupervised TSDN / GPG)
3. Co‐saliency map generation via cosal.py (CMP module)
4. Evaluation via evaluate.py (F-measure, S-measure, MAE)
• Data directories: /data/CoSal2015, CoCA, CoSOD3k containing image groups, captions, and ground truths.
Main components & interactions
a. Components:
• Input Data Store (image groups, ground truths, captions)
• CLI Orchestrator (shell scripts + main.py / cosal.py)
• Feature Extraction Module (sd-dino)
• Saliency Module (A2S-v2: GPG + TSDN)
• Co‐Saliency Module (CMP)
• Segmentation Model (SAM via segment_anything)
• Evaluation Module (evaluate.py + CoSOD toolbox)
• Configuration Layer (YAML/JSON configs under sd-dino/dinov2/configs & A2S-v2/base/config.py & methods/*/config.py)
• Logging & Utilities (utils.py across modules)
b. Relationships & data flow:
1. User triggers “sh sd-dino/extract_feat.sh” → CLI calls sd-dino/scripts → Python feature extractor → writes feature tensors to disk
2. CLI calls “sh A2S-v2/inference_sod.sh” → loads features and A2S-v2/base/model → runs GPG → produces saliency masks → writes SOD outputs
3. CLI calls “sh inference_cosod.sh” → cosal.py loads saliency masks + features + prompts → CMP computes co‐saliency maps → writes results to /results
4. User runs “python evaluate.py” → loads predictions + ground truth → computes metrics → outputs quantitative scores
c. Architectural patterns:
• Modular directory layout by functional concern (feature‐extractor, saliency, co‐saliency, evaluation)
• Factory pattern in base/framework_factory.py to instantiate encoders and models dynamically
• Shell‐script orchestration (batch pipeline) rather than real‐time service
• Config‐driven (YAML & Python) for both training/inference settings
d. Technologies & frameworks:
• Python 3.9, PyTorch, Conda environments
• Shell scripts (Bash) for orchestration
• Stable Diffusion, DINOv2, Segment Anything Model (SAM), TSDN
• Lightweight Python modules for utilities, metrics (F-measure, MAE, S-measure)
Capturing essential details
• Do not over‐compartmentalize; stick to the major pipeline stages and their submodules
• Component’s technology/framework is visible
• How the Group Prompt Generation (GPG) feeds into TSDN saliency, and how CMP merges results