# uark-aicv/gazesearch architecture

> Architecture diagram of the GitHub repository uark-aicv/gazesearch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/uark-aicv/gazesearch
- Repository: https://github.com/uark-aicv/gazesearch

## Overview

##### Project Type and Purpose
- This is a research-focused full-stack “benchmark + baseline model” project.
- Purpose: provide a curated eye-tracking dataset (GazeSearch) and a scanpath-prediction model (ChestSearch) for medical image analysis.

##### File Structure
Top-level grouping:
- data/ – processed COCO-format JSON of gaze data
- example/, imgs/ – sample inputs and outputs for demo
- src/ – all Python source code
– chestsearch/ – core model code (backbone, pixel/transformer decoders, builder, config, evaluation, inference)
– common/ – shared utilities (data loaders, metrics, losses, sinkhorn, environment, etc.)
– configs/ – JSON/YAML model and eval configs
– inference/ – output folders for model predictions
– demo_medical.ipynb – end-to-end interactive demo
– requirements.txt, INSTALL.md – setup

##### README for Additional Insights
- ChestSearch is the baseline model, built around a configurable builder pattern.
- It uses PyTorch+CUDA custom ops (pixel_decoder/ops) and transformer modules.
- Demo notebook illustrates data→model→evaluation flow.

##### Draft the Ma
a. Data Layer (storage & access)
– data/finding_visual_search_…json (COCO-formatted gaze sequences)
– example images in example/ & imgs/ (for qualitative visualization)
b. Configuration Layer
– configs/*.json & resnet50.yaml drive hyperparameters, model architecture
c. Preprocessing & Dataset APIs
– common/dataset.py, coco_det.py, inference_dataset.py
– common/data.py loads images + gaze points
d. Model Construction (Builder Pattern)
– chestsearch/builder.py & inference_builder.py: assemble backbone + pixel_decoder + transformer_decoder → full model
e. Backbone & Feature Extraction
– chestsearch/backbone/swin.py (Swin Transformer)
f. Pixel Decoder + Custom CUDA Ops
– chestsearch/pixel_decoder/fpn.py
– chestsearch/pixel_decoder/msdeformattn.py
– pixel_decoder/ops: setup, C++/CUDA kernels for multi-scale deformable attention
g. Transformer Decoder
– chestsearch/transformer_decoder/… (mask2former style multi-head attention + position_encoding)
h. Training & Evaluation
– chestsearch/config.py loads config
– chestsearch/evaluation.py & inference_evaluation.py compute metrics
– common/losses.py, common/metrics.py, common/visual_attention_metrics.py
i. Inference Pipeline
– inference_builder.py + inference_dataset.py → chestsearch/inference outputs (scanpath.json)
j. Demo & Visualization
– src/demo_medical.ipynb uses matplotlib/seaborn + example images

##### Patterns and Principles
- Builder pattern for model construction (clear separation of backbone, decoders).
- Modular custom-ops for performance-critical deformable attention.
- Config-driven design for experiment reproducibility.
- Clear separation of “common” utilities vs. experiment-specific code.

##### Final Checklist
[ ] Each major component labeled (e.g., “Backbone: Swin Transformer,” “Pixel Decoder + CUDA Ops”)
[ ] Config files shown feeding into builder modules
[ ] Group boundaries (e.g., a “src” container showing subpackages common/ and chestsearch/)
[ ] Demo notebook at top-level showing end-to-end orchestration

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Demo Layer
    Demo["demo_medical.ipynb"]:::demo
    click Demo "https://github.com/uark-aicv/gazesearch/blob/main/src/demo_medical.ipynb"

    %% Data Layer
    subgraph "Data Layer"
        direction TB
        DataJSON(("GazeSearch JSON")):::data
        ExampleImgs(("Example & imgs")):::data
    end
    click DataJSON "https://github.com/uark-aicv/gazesearch/blob/main/data/finding_visual_search_coco_format_train_test_filtered_max_6_split_train_valid_test_2024-07-22.json"
    click ExampleImgs "https://github.com/uark-aicv/gazesearch/tree/main/example/"
    click ExampleImgs "https://github.com/uark-aicv/gazesearch/tree/main/imgs/"

    %% Config Layer
    subgraph "Configuration Layer"
        direction TB
        Config1["finding_search_box_mask_max6_split_shuffled.json"]:::config
        Config2["eval_finding_search_box_mask_max6_split.json"]:::config
        Config3["resnet50.yaml"]:::config
    end
    click Config1 "https://github.com/uark-aicv/gazesearch/blob/main/src/configs/finding_search_box_mask_max6_split_shuffled.json"
    click Config2 "https://github.com/uark-aicv/gazesearch/blob/main/src/configs/eval_finding_search_box_mask_max6_split.json"
    click Config3 "https://github.com/uark-aicv/gazesearch/blob/main/src/configs/resnet50.yaml"

    %% Source Code Layer
    subgraph "Source Code (src)"
        direction TB

        subgraph common [common]
            direction TB
            DataAPI["data.py"]:::util
            Dataset["dataset.py"]:::util
            CocoDet["coco_det.py"]:::util
            InferenceDS["inference_dataset.py"]:::util
            Losses["losses.py"]:::util
            Metrics["metrics.py"]:::util
            VisMetrics["visual_attention_metrics.py"]:::util
        end
        click DataAPI "https://github.com/uark-aicv/gazesearch/blob/main/src/common/data.py"
        click Dataset "https://github.com/uark-aicv/gazesearch/blob/main/src/common/dataset.py"
        click CocoDet "https://github.com/uark-aicv/gazesearch/blob/main/src/common/coco_det.py"
        click InferenceDS "https://github.com/uark-aicv/gazesearch/blob/main/src/common/inference_dataset.py"
        click Losses "https://github.com/uark-aicv/gazesearch/blob/main/src/common/losses.py"
        click Metrics "https://github.com/uark-aicv/gazesearch/blob/main/src/common/metrics.py"
        click VisMetrics "https://github.com/uark-aicv/gazesearch/blob/main/src/common/visual_attention_metrics.py"

        subgraph chestsearch [chestsearch]
            direction TB
            Builder["builder.py"]:::model
            InferenceBuilder["inference_builder.py"]:::model
            ConfigLoader["config.py"]:::model
            Evaluation["evaluation.py"]:::train
            InferenceEval["inference_evaluation.py"]:::train

            subgraph Backbone
                Swin["swin.py"]:::model
            end
            click Swin "https://github.com/uark-aicv/gazesearch/blob/main/src/chestsearch/backbone/swin.py"

            subgraph PixelDecoder
                FPN["fpn.py"]:::model
                MSDefAttn["msdeformattn.py"]:::model
                subgraph "ops (CUDA)"
                    direction TB
                    OpsModules["ops/"]:::model
                end
            end
            click FPN "https://github.com/uark-aicv/gazesearch/blob/main/src/chestsearch/pixel_decoder/fpn.py"
            click MSDefAttn "https://github.com/uark-aicv/gazesearch/blob/main/src/chestsearch/pixel_decoder/msdeformattn.py"
            click OpsModules "https://github.com/uark-aicv/gazesearch/tree/main/src/chestsearch/pixel_decoder/ops/"

            subgraph TransformerDecoder
                Mask2Former["mask2former_transformer_decoder.py"]:::model
                PosEnc["position_encoding.py"]:::model
                Transf["transformer.py"]:::model
            end
            click Mask2Former "https://github.com/uark-aicv/gazesearch/blob/main/src/chestsearch/transformer_decoder/mask2former_transformer_decoder.py"
            click PosEnc "https://github.com/uark-aicv/gazesearch/blob/main/src/chestsearch/transformer_decoder/position_encoding.py"
            click Transf "https://github.com/uark-aicv/gazesearch/blob/main/src/chestsearch/transformer_decoder/transformer.py"
        end
        click Builder "https://github.com/uark-aicv/gazesearch/blob/main/src/chestsearch/builder.py"
        click InferenceBuilder "https://github.com/uark-aicv/gazesearch/blob/main/src/chestsearch/inference_builder.py"
        click ConfigLoader "https://github.com/uark-aicv/gazesearch/blob/main/src/chestsearch/config.py"
        click Evaluation "https://github.com/uark-aicv/gazesearch/blob/main/src/chestsearch/evaluation.py"
        click InferenceEval "https://github.com/uark-aicv/gazesearch/blob/main/src/chestsearch/inference_evaluation.py"
    end

    %% Inference Output
    Scanpath(("scanpath.json")):::data
    click Scanpath "https://github.com/uark-aicv/gazesearch/blob/main/src/inference/eval_finding_search_box_mask_max6_split/scanpath.json"

    %% Connections
    Config1 -->|controls| Builder
    Config2 -->|controls| Evaluation
    Config3 -->|defines| Builder

    DataJSON -->|loads| DataAPI
    ExampleImgs -->|images| DataAPI
    DataAPI -->|produce| Dataset
    Dataset -->|feeds| Builder

    Builder -->|builds| Swin
    Builder -->|builds| PixelDecoder
    Builder -->|builds| TransformerDecoder
    Builder -->|instantiates| ConfigLoader

    Swin -->|features| PixelDecoder
    PixelDecoder -->|features| TransformerDecoder

    TransformerDecoder -->|forward| Evaluation
    Losses -->|compute| Evaluation
    Metrics -->|compute| Evaluation
    VisMetrics -->|compute| Evaluation

    InferenceDS -->|loads| InferenceBuilder
    InferenceBuilder -->|model| InferenceEval
    InferenceEval -->|outputs| Scanpath

    Demo -->|uses| Dataset
    Demo -->|uses| Builder
    Demo -->|uses| Evaluation
    Demo -->|visualizes| ExampleImgs

    %% Styles
    classDef data fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef util fill:#90EE90,stroke:#000,stroke-width:1px
    classDef model fill:#FFA500,stroke:#000,stroke-width:1px
    classDef train fill:#DDA0DD,stroke:#000,stroke-width:1px
    classDef demo fill:#D3D3D3,stroke:#000,stroke-width:1px
    classDef config fill:#FFB6C1,stroke:#000,stroke-width:1px
```
