# hkxiao/zs-cosod architecture

> Architecture diagram of the GitHub repository hkxiao/zs-cosod, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hkxiao/zs-cosod
- Repository: https://github.com/hkxiao/zs-cosod

## Overview

##### Project type and purpose
- This is a research‐oriented Python deep‐learning pipeline for zero‐shot Co‐salient Object Detection (CoSOD). It is not a web or full‐stack app but a modular ML framework that leverages multiple open‐source models (Stable Diffusion, DINOv2, SAM, TSDN) to perform a series of data‐processing and inference steps without any training.
- The README describes two core algorithmic components: the Group Prompt Generation (GPG) module and the Co‐Saliency Map Generation (CMP) module, plus feature‐extraction and evaluation steps.

##### File structure
- Top‐level modules:
- sd-dino/: Stable Diffusion + DINOv2‐based feature extractor
- A2S-v2/: TSDN unsupervised saliency model (config, encoder, method factories)
- segment_anything/: SAM model and utilities
- cosal.py, main.py, evaluate.py, etc.: orchestration scripts for full CoSOD pipeline
- Shell scripts in root and subdirectories (extract_feat.sh, inference_sod.sh, inference_cosod.sh) orchestrate the flow.
- Utility modules (util.py, data.py, framework_factory.py) provide shared functions and factory patterns for model instantiation.
- No web server, database, or frontend; it’s a batch inference and evaluation pipeline.

##### Key insights from README
- Workflow:
1. Feature extraction via sd-dino (Stable Diffusion + DINOv2)
2. Saliency map generation via A2S-v2 (Unsupervised TSDN / GPG)
3. Co‐saliency map generation via cosal.py (CMP module)
4. Evaluation via evaluate.py (F-measure, S-measure, MAE)
- Data directories: /data/CoSal2015, CoCA, CoSOD3k containing image groups, captions, and ground truths.

##### Main components & interactions
a. Components:
- Input Data Store (image groups, ground truths, captions)
- CLI Orchestrator (shell scripts + main.py / cosal.py)
- Feature Extraction Module (sd-dino)
- Saliency Module (A2S-v2: GPG + TSDN)
- Co‐Saliency Module (CMP)
- Segmentation Model (SAM via segment_anything)
- Evaluation Module (evaluate.py + CoSOD toolbox)
- Configuration Layer (YAML/JSON configs under sd-dino/dinov2/configs & A2S-v2/base/config.py & methods/*/config.py)
- Logging & Utilities (utils.py across modules)
b. Relationships & data flow:
1. User triggers “sh sd-dino/extract_feat.sh” → CLI calls sd-dino/scripts → Python feature extractor → writes feature tensors to disk
2. CLI calls “sh A2S-v2/inference_sod.sh” → loads features and A2S-v2/base/model → runs GPG → produces saliency masks → writes SOD outputs
3. CLI calls “sh inference_cosod.sh” → cosal.py loads saliency masks + features + prompts → CMP computes co‐saliency maps → writes results to /results
4. User runs “python evaluate.py” → loads predictions + ground truth → computes metrics → outputs quantitative scores
c. Architectural patterns:
- Modular directory layout by functional concern (feature‐extractor, saliency, co‐saliency, evaluation)
- Factory pattern in base/framework_factory.py to instantiate encoders and models dynamically
- Shell‐script orchestration (batch pipeline) rather than real‐time service
- Config‐driven (YAML & Python) for both training/inference settings
d. Technologies & frameworks:
- Python 3.9, PyTorch, Conda environments
- Shell scripts (Bash) for orchestration
- Stable Diffusion, DINOv2, Segment Anything Model (SAM), TSDN
- Lightweight Python modules for utilities, metrics (F-measure, MAE, S-measure)

##### Capturing essential details
- Do not over‐compartmentalize; stick to the major pipeline stages and their submodules
- Component’s technology/framework is visible
- How the Group Prompt Generation (GPG) feeds into TSDN saliency, and how CMP merges results

## Diagram (Mermaid)

```mermaid
flowchart TB
    classDef data fill:#bbdefb,stroke:#1e88e5,color:#0d47a1
    classDef process fill:#c8e6c9,stroke:#388e3c,color:#1b5e20
    classDef model fill:#ffcc80,stroke:#ef6c00,color:#e65100
    classDef cli fill:#eeeeee,stroke:#9e9e9e,color:#424242
    classDef config fill:#e1bee7,stroke:#8e24aa,color:#4a148c
    classDef util fill:#cfcccc,stroke:#616161,color:#212121

    subgraph "Data Stores"
        DS1["Raw Image Groups\n/data/CoSal..."]:::data
        DS2["Feature Tensors\n/features/"]:::data
        DS3["Saliency Masks\n/sod_outputs/"]:::data
        DS4["Co-Saliency Maps\n/results/"]:::data
        DS5["Ground Truth & Captions"]:::data
    end

    subgraph "CLI Orchestrator"
        CF1["extract_feat.sh"]:::cli
        CF2["inference_sod.sh"]:::cli
        CF3["inference_cosod.sh"]:::cli
        CF4["eval_cosod.sh"]:::cli
        CF5["main.py"]:::cli
        CF6["cosal.py"]:::cli
        CF7["evaluate.py"]:::cli
    end

    subgraph "Feature Extraction Module"
        FEM1["extractor_sd.py"]:::process
        FEM2["extractor_dino.py"]:::process
        FEM3["extract_sd_raw+dino.py"]:::process
    end

    subgraph "Saliency Module (A2S-v2)"
        SM1["base/model.py"]:::process
        SM2["base/encoder/"]:::model
        SM3["methods/a2s (GPG)"]:::process
    end

    subgraph "Segmentation Model (SAM)"
        SAM1["automatic_mask_generator.py"]:::model
        SAM2["predictor.py"]:::model
        SAM3["modeling/sam.py"]:::model
    end

    subgraph "Co-Saliency Module (CMP)"
        CMP1["cosal.py"]:::process
        CMP2["process_feat.py"]:::process
    end

    subgraph "Evaluation Module"
        EM1["evaluate.py"]:::process
        EM2["evaluator/evaluator.py"]:::process
        EM3["evaluator/fmeasure.py"]:::process
        EM4["evaluator/mae.py"]:::process
        EM5["evaluator/smeasure.py"]:::process
    end

    subgraph "Configuration Layer"
        CFG1["dinov2/configs/"]:::config
        CFG2["A2S-v2/base/config.py"]:::config
        CFG3["methods/*/config.py"]:::config
    end

    subgraph "Logging & Utilities"
        UT1["sd-dino/utils/logger.py"]:::util
        UT2["A2S-v2/util.py"]:::util
        UT3["A2S-v2/base/framework_factory.py"]:::util
        UT4["kmeans_pytorch.py"]:::util
        UT5["util.py"]:::util
    end

    DS1 -->|"images"| CF1
    CF1 --> FEM1
    CF1 --> FEM2
    FEM1 -->|" .pt tensors"| DS2
    FEM2 --> DS2

    DS2 -->|"features"| CF2
    CF2 --> SM1
    CF2 --> SM3
    SM1 -->|"masks"| DS3
    SM3 --> DS3

    DS2 --> CF3
    DS3 --> CF3
    CF3 --> CMP2
    CMP2 --> CMP1
    CMP1 -->|"maps"| DS4

    DS4 --> CF4
    DS5 --> CF4
    CF4 --> EM1
    EM1 --> EM2
    EM1 --> EM3
    EM1 --> EM4
    EM1 --> EM5

    CFG1 --- FEM1
    CFG2 --- SM1
    CFG3 --- CMP1
    UT1 --- FEM1
    UT2 --- SM1
    UT3 --- SM1
    UT3 --- CMP1
    UT4 --- CMP2
    UT5 --- EM1

    click CF1 "https://github.com/hkxiao/zs-cosod/blob/master/sd-dino/extract_feat.sh"
    click FEM1 "https://github.com/hkxiao/zs-cosod/blob/master/sd-dino/extractor_sd.py"
    click FEM2 "https://github.com/hkxiao/zs-cosod/blob/master/sd-dino/extractor_dino.py"
    click FEM3 "https://github.com/hkxiao/zs-cosod/blob/master/sd-dino/extract_sd_raw+dino.py"
    click SM1 "https://github.com/hkxiao/zs-cosod/blob/master/A2S-v2/base/model.py"
    click SM2 "https://github.com/hkxiao/zs-cosod/tree/master/A2S-v2/base/encoder/"
    click SM3 "https://github.com/hkxiao/zs-cosod/tree/master/A2S-v2/methods/a2s"
    click CF2 "https://github.com/hkxiao/zs-cosod/blob/master/A2S-v2/inference_sod.sh"
    click CF3 "https://github.com/hkxiao/zs-cosod/blob/master/inference_cosod.sh"
    click CF4 "https://github.com/hkxiao/zs-cosod/blob/master/eval_cosod.sh"
    click CF5 "https://github.com/hkxiao/zs-cosod/blob/master/main.py"
    click CF6 "https://github.com/hkxiao/zs-cosod/blob/master/cosal.py"
    click CF7 "https://github.com/hkxiao/zs-cosod/blob/master/evaluate.py"
    click SAM1 "https://github.com/hkxiao/zs-cosod/blob/master/segment_anything/automatic_mask_generator.py"
    click SAM2 "https://github.com/hkxiao/zs-cosod/blob/master/segment_anything/predictor.py"
    click SAM3 "https://github.com/hkxiao/zs-cosod/blob/master/segment_anything/modeling/sam.py"
    click CFG1 "https://github.com/hkxiao/zs-cosod/tree/master/sd-dino/dinov2/dinov2/configs/"
    click CFG2 "https://github.com/hkxiao/zs-cosod/blob/master/A2S-v2/base/config.py"
    click CFG3 "https://github.com/hkxiao/zs-cosod/blob/master/A2S-v2/methods/*/config.py"
    click UT1 "https://github.com/hkxiao/zs-cosod/blob/master/sd-dino/utils/logger.py"
    click UT2 "https://github.com/hkxiao/zs-cosod/blob/master/A2S-v2/util.py"
    click UT3 "https://github.com/hkxiao/zs-cosod/blob/master/A2S-v2/base/framework_factory.py"
    click UT4 "https://github.com/hkxiao/zs-cosod/blob/master/kmeans_pytorch.py"
    click UT5 "https://github.com/hkxiao/zs-cosod/blob/master/util.py"
```
