How 7hhhhh/visualad works
1. Repository purpose and execution flow
- This is a PyTorch repo for zero-shot anomaly detection with a frozen Vision Transformer backbone.
- The main runtime entry points are train.py and test.py, with scripts/CLIP.sh orchestrating the common cross-dataset pipeline used in the README.
- Pretrained checkpoints live under weight/, so the repo supports both training and direct evaluation.
2. Data preparation boundary
- Raw datasets are not bundled here; the repo expects the same folder/JSON convention as AnomalyCLIP.
- generate_dataset_json/ contains one script per benchmark dataset and is responsible for producing dataset metadata JSON files.
- dataset.py is the bridge from those JSON descriptors to PyTorch dataset objects consumed by training and evaluation.
3. Core model package
- VisualAD_lib/VisualAD.py appears to define the main VisualAD model.
- build_model.py assembles the backbone, token logic, and anomaly-scoring components.
- model_load.py handles checkpoint loading and model restoration, which matters for cross-dataset evaluation.
- constants.py centralizes model- and dataset-related settings, while __init__.py exposes the package API.
4. Text-free tokenized ViT architecture
- The model is built around a frozen ViT backbone, with two learnable tokens: one representing normality and one representing anomaly.
- The repo is explicitly language-free, so there is no text encoder path; all reasoning happens in visual feature space.
- VisualAD_lib/simple_tokenizer.py, bpe_simple_vocab_16e6.txt.gz, and transform.py are present because the codebase is CLIP-compatible, even though VisualAD removes the text branch.
5. Backbone adaptation layer
- utils/backbone_config.py is the main boundary for supporting multiple backbones.
- The README indicates two supported backbones: CLIP ViT-L/14@336px and DINOv2 ViT-L/14.
- This layer is where backbone-specific feature shapes, preprocessing, and loading behavior are normalized so the rest of the model can stay shared.
6. Feature interaction modules
- utils/spatial_cross_attention.py corresponds to the Spatial-Aware Cross-Attention idea described in the README.
- utils/feature_transform.py, utils/normalization.py, and utils/transforms.py likely handle token/patch feature reshaping, recalibration, and preprocessing.
- utils/anomaly_detection.py and utils/scoring.py appear to convert refined features into image-level and pixel-level anomaly scores.
7. Training subsystem
- train.py likely drives the standard optimization loop using utils/training_utils.py.
- utils/loss.py defines the training objectives for separating normal and anomalous representations.
- utils/optim/ contains optimizer setup, while utils/logger.py handles experiment logging and utils/metrics.py tracks performance during training/validation.
8. Inference and evaluation subsystem
- test.py is the evaluation entry point, using trained or pretrained weights to produce anomaly predictions.
- Evaluation output is fed through utils/metrics.py for image-level AUROC/F1/AP and pixel-level AUROC/F1/AP/PRO, matching the metrics shown in the README.
- utils/visualization.py supports qualitative inspection of anomaly heatmaps and masks.
9. Analysis and debugging support
- utils/analysis.py suggests some offline analysis of scores, features, or failure cases.
- These utilities are separate from the core training loop, indicating the repo keeps experimentation/diagnostics out of the model path.
10. Dataset coverage and cross-domain design
- The codebase is organized around multiple industrial and medical benchmarks: MVTec, VisA, BTAD, KSDD2, DAGM, DTD-Synthetic, OCT17, BrainMRI, Brain_AD, HIS, CVC-ClinicDB, Endo, and Kvasir.
- The architecture is designed for cross-dataset zero-shot transfer: train on one source dataset, evaluate on another without task-specific adaptation.
- The presence of per-dataset JSON generators and shared dataset loading suggests a uniform dataset contract across very different domains.
11. External dependencies and runtime boundary
- The main stack is PyTorch with torchvision and timm, plus scikit-learn, scipy, and tqdm for metrics and utilities.
- Checkpoints under weight/train_on_mvtec/CLIP.pth and weight/train_on_visa/CLIP.pth are the repo’s shipped model artifacts.
- The project depends on external backbone ecosystems, especially CLIP and DINOv2, but wraps them so the anomaly-detection logic remains backbone-agnostic.
12. How data flows through the system
- Dataset JSON and image folders are read by dataset.py, transformed, and batched into the training/eval loop.
- Batches pass through the frozen ViT backbone and the learnable token modules in VisualAD_lib.
- Spatial and feature recalibration modules refine patch-level evidence before scoring.
- Final outputs are anomaly scores and heatmaps, which are then measured by the metrics and optionally visualized.
Main entry points: train.py train.py · test.py test.py · VisualAD.py VisualAD · dataset.py dataset.py
Components 20
Entry points
Training, evaluation, and orchestration scripts.
Data pipeline
Dataset contract and benchmark JSON generators.
- dataset.py · Converts dataset JSON into PyTorch datasets.
dataset.py - JSON generators · Per-benchmark scripts that build dataset descriptors.
generate_dataset_json
VisualAD core
Model package, backbone adaptation, and checkpoint loading.
- VisualAD_lib · Core VisualAD implementation and package API.
VisualAD_lib - VisualAD · Defines the tokenized zero-shot anomaly detector.
VisualAD_lib/VisualAD.py - build_model.py · Constructs backbone, tokens, and scoring path.
VisualAD_lib/build_model.py - model_load.py · Restores pretrained or trained weights.
VisualAD_lib/model_load.py
Runtime utils
Training, scoring, metrics, visualization, and analysis support.
- backbone_config.py · Normalizes CLIP and DINOv2 backbone behavior.
utils/backbone_config.py - spatial attention · Spatial-aware cross-attention refinement.
utils/spatial_cross_attention.py - feature ops · Feature reshaping and recalibration helpers.
utils/feature_transform.py - scoring · Converts refined features into anomaly scores.
utils/scoring.py - metrics · Computes AUROC, AP, F1, and PRO-style scores.
utils/metrics.py - training utils · Shared training loop helpers and batch handling.
utils/training_utils.py - loss.py · Training objectives for normal/anomalous separation.
utils/loss.py - visualization · Heatmap and mask visualization utilities.
utils/visualization.py - analysis · Offline score and failure-case analysis.
utils/analysis.py
Connections 24
- CLIP.sh → train.py: runs · Common training pipeline entry.
- CLIP.sh → test.py: evaluates · Common evaluation pipeline entry.
- train.py → dataset.py: loads · Reads dataset JSON and image folders.
- test.py → dataset.py: loads · Builds evaluation datasets from the same contract.
- dataset.py → JSON generators: expects · Consumes benchmark-specific JSON metadata.
- train.py → VisualAD_lib: trains · Optimizes the core VisualAD model.
- test.py → VisualAD_lib: inferences · Runs the model for anomaly prediction.
- VisualAD_lib → VisualAD: contains · Package exposes the main model implementation.
- VisualAD → build_model.py: assembled by · Model construction is centralized here.
- build_model.py → backbone_config.py: adapts · Selects and normalizes supported backbones.
- VisualAD → spatial attention: refines · Applies spatial-aware feature interaction.
- VisualAD → feature ops: transforms · Shapes token and patch features for scoring.
- VisualAD → scoring: produces · Outputs anomaly scores and maps.
- model_load.py → VisualAD: restores · Loads checkpoints into the model.
- train.py → training utils: uses · Shared optimization and batching helpers.
- train.py → loss.py: optimizes · Training objective for separation.
- test.py → metrics: measures · Computes evaluation metrics.
- train.py → metrics: tracks · Monitors validation performance.
- test.py → visualization: renders · Supports qualitative inspection.
- analysis → scoring: inspects · Offline analysis of scores and failures.
- train.py → CLIP.pth: reads · Can initialize from shipped checkpoint.
- train.py → CLIP.pth: reads · Can initialize from shipped checkpoint.
- test.py → CLIP.pth: evaluates · Direct evaluation from pretrained weights.
- test.py → CLIP.pth: evaluates · Direct evaluation from pretrained weights.