How sxing2/clip-test-time-counterattacks works
Project Type and Purpose
• This is a research‐code repository (“CLIP-Test‐time‐Counterattacks”) implementing a test‐time defense paradigm (TTC) for zero‐shot robustness of the CLIP vision‐language model against adversarial attacks.
• It is neither a web app nor a microservices system—it’s a Python‐based machine‐learning pipeline orchestrated via scripts.
File‐Structure Analysis
• Top‐level directories/files
– code/: core Python modules (attacks, model wrappers, counterattack methods, utilities)
– AFT_model_weights/: pretrained adversarially‐finetuned model checkpoints
– data/: raw datasets or placeholders for torchvision downloads
– support/: label mappings and auxiliary metadata
– figures/: static images (e.g., poster)
– TTC.sh, download_weights.py, test_time_counterattack.py: entrypoint scripts
– environment.yml, requirements.txt: environment specification
• Recognized patterns
– Modular design: attack routines (code/attacks.py), model interfaces (code/models/), replacement/counterattack logic (code/replace/)
– Single‐stage pipeline: load data → generate adversarial examples → apply TTC counterattack → evaluate → save results
Key README Insights
• Entry point: code/test_time_counterattack.py (or TTC.sh wrapper)
• Two modes: original CLIP or adversarially fine‐tuned model (from AFT_model_weights/)
• Configuration via CLI args (batch_size, attack type, eps, steps, etc.)
• Dependencies: PyTorch, torchvision, OpenAI’s CLIP library
a. Main Components
1. Script Launcher (bash TTC.sh or python test_time_counterattack.py)
2. Configuration & CLI Parser
3. Dataset Loader (torchvision.datasets or custom wrappers under code/replace/datasets/)
4. Model Manager
• CLIP vision‐encoder (code/models/model.py)
• Prompt engineering module (code/models/prompters.py)
• Optional adversarially fine‐tuned weights loader (downloaded into AFT_model_weights/)
5. Attack Generator (code/attacks.py): PGD, FGSM, etc.
6. TTC Counterattack Module (code/replace/clip.py + code/replace/model.py + simple_tokenizer)
7. Evaluation & Metrics Collector (code/utils.py)
8. Result Saver (writes to --outdir, e.g., TTC_results/)
b. Relationships & Data Flow
• Launcher passes CLI args to Config Parser
• Config instantiates Dataset Loader → yields batches of images & labels
• Each batch feeds into Attack Generator → adversarial images
• Adversarial images enter TTC Counterattack Module → “corrected” images
• Corrected images + textual prompts go into Model Manager → class scores
• Metrics Collector computes accuracy/robustness statistics
• Result Saver writes logs, numeric results, possibly figures
c. Architectural Patterns & Principles
• Pipeline pattern: linear flow with modular stages
• Separation of concerns: attack logic, counterattack logic, model abstraction, I/O
d. Key Technologies & Libraries
• Python 3, PyTorch, torchvision
• OpenAI’s CLIP library
• Numpy, tqdm (progress bars), argparse (CLI), optional datasets auto‐download