How uark-aicv/gazesearch works
Project Type and Purpose
• This is a research-focused full-stack “benchmark + baseline model” project.
• Purpose: provide a curated eye-tracking dataset (GazeSearch) and a scanpath-prediction model (ChestSearch) for medical image analysis.
File Structure
Top-level grouping:
• data/ – processed COCO-format JSON of gaze data
• example/, imgs/ – sample inputs and outputs for demo
• src/ – all Python source code
– chestsearch/ – core model code (backbone, pixel/transformer decoders, builder, config, evaluation, inference)
– common/ – shared utilities (data loaders, metrics, losses, sinkhorn, environment, etc.)
– configs/ – JSON/YAML model and eval configs
– inference/ – output folders for model predictions
– demo_medical.ipynb – end-to-end interactive demo
– requirements.txt, INSTALL.md – setup
README for Additional Insights
• ChestSearch is the baseline model, built around a configurable builder pattern.
• It uses PyTorch+CUDA custom ops (pixel_decoder/ops) and transformer modules.
• Demo notebook illustrates data→model→evaluation flow.
Draft the Ma
a. Data Layer (storage & access)
– data/finding_visual_search_…json (COCO-formatted gaze sequences)
– example images in example/ & imgs/ (for qualitative visualization)
b. Configuration Layer
– configs/*.json & resnet50.yaml drive hyperparameters, model architecture
c. Preprocessing & Dataset APIs
– common/dataset.py, coco_det.py, inference_dataset.py
– common/data.py loads images + gaze points
d. Model Construction (Builder Pattern)
– chestsearch/builder.py & inference_builder.py: assemble backbone + pixel_decoder + transformer_decoder → full model
e. Backbone & Feature Extraction
– chestsearch/backbone/swin.py (Swin Transformer)
f. Pixel Decoder + Custom CUDA Ops
– chestsearch/pixel_decoder/fpn.py
– chestsearch/pixel_decoder/msdeformattn.py
– pixel_decoder/ops: setup, C++/CUDA kernels for multi-scale deformable attention
g. Transformer Decoder
– chestsearch/transformer_decoder/… (mask2former style multi-head attention + position_encoding)
h. Training & Evaluation
– chestsearch/config.py loads config
– chestsearch/evaluation.py & inference_evaluation.py compute metrics
– common/losses.py, common/metrics.py, common/visual_attention_metrics.py
i. Inference Pipeline
– inference_builder.py + inference_dataset.py → chestsearch/inference outputs (scanpath.json)
j. Demo & Visualization
– src/demo_medical.ipynb uses matplotlib/seaborn + example images
Patterns and Principles
• Builder pattern for model construction (clear separation of backbone, decoders).
• Modular custom-ops for performance-critical deformable attention.
• Config-driven design for experiment reproducibility.
• Clear separation of “common” utilities vs. experiment-specific code.
Final Checklist
[ ] Each major component labeled (e.g., “Backbone: Swin Transformer,” “Pixel Decoder + CUDA Ops”)
[ ] Config files shown feeding into builder modules
[ ] Group boundaries (e.g., a “src” container showing subpackages common/ and chestsearch/)
[ ] Demo notebook at top-level showing end-to-end orchestration