# sxing2/clip-test-time-counterattacks architecture

> Architecture diagram of the GitHub repository sxing2/clip-test-time-counterattacks, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sxing2/clip-test-time-counterattacks
- Repository: https://github.com/sxing2/clip-test-time-counterattacks

## Overview

##### Project Type and Purpose
- This is a research‐code repository (“CLIP-Test‐time‐Counterattacks”) implementing a test‐time defense paradigm (TTC) for zero‐shot robustness of the CLIP vision‐language model against adversarial attacks.
- It is neither a web app nor a microservices system—it’s a Python‐based machine‐learning pipeline orchestrated via scripts.

##### File‐Structure Analysis
- Top‐level directories/files
– code/: core Python modules (attacks, model wrappers, counterattack methods, utilities)
– AFT_model_weights/: pretrained adversarially‐fine­tuned model checkpoints
– data/: raw datasets or placeholders for torchvision downloads
– support/: label mappings and auxiliary metadata
– figures/: static images (e.g., poster)
– TTC.sh, download_weights.py, test_time_counterattack.py: entrypoint scripts
– environment.yml, requirements.txt: environment specification
- Recognized patterns
– Modular design: attack routines (code/attacks.py), model interfaces (code/models/), replacement/counterattack logic (code/replace/)
– Single‐stage pipeline: load data → generate adversarial examples → apply TTC counterattack → evaluate → save results

##### Key README Insights
- Entry point: code/test_time_counterattack.py (or TTC.sh wrapper)
- Two modes: original CLIP or adversarially fine‐tuned model (from AFT_model_weights/)
- Configuration via CLI args (batch_size, attack type, eps, steps, etc.)
- Dependencies: PyTorch, torchvision, OpenAI’s CLIP library

a. Main Components
1. Script Launcher (bash TTC.sh or python test_time_counterattack.py)
2. Configuration & CLI Parser
3. Dataset Loader (torchvision.datasets or custom wrappers under code/replace/datasets/)
4. Model Manager
- CLIP vision‐encoder (code/models/model.py)
- Prompt engineering module (code/models/prompters.py)
- Optional adversarially fine‐tuned weights loader (downloaded into AFT_model_weights/)
5. Attack Generator (code/attacks.py): PGD, FGSM, etc.
6. TTC Counterattack Module (code/replace/clip.py + code/replace/model.py + simple_tokenizer)
7. Evaluation & Metrics Collector (code/utils.py)
8. Result Saver (writes to --outdir, e.g., TTC_results/)
b. Relationships & Data Flow
- Launcher passes CLI args to Config Parser
- Config instantiates Dataset Loader → yields batches of images & labels
- Each batch feeds into Attack Generator → adversarial images
- Adversarial images enter TTC Counterattack Module → “corrected” images
- Corrected images + textual prompts go into Model Manager → class scores
- Metrics Collector computes accuracy/robustness statistics
- Result Saver writes logs, numeric results, possibly figures
c. Architectural Patterns & Principles
- Pipeline pattern: linear flow with modular stages
- Separation of concerns: attack logic, counterattack logic, model abstraction, I/O
d. Key Technologies & Libraries
- Python 3, PyTorch, torchvision
- OpenAI’s CLIP library
- Numpy, tqdm (progress bars), argparse (CLI), optional datasets auto‐download

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Entry and orchestration
    ScriptLauncher["Script Launcher\n(TTC.sh / test_time_counterattack.py)"]:::orchestration
    ConfigParser["Configuration & CLI Parser"]:::orchestration
    ResultSaver["Result Saver\n(writes to --outdir)"]:::orchestration

    %% Data components
    DatasetLoader["Dataset Loader"]:::data
    TTCResults[(TTC_results folder)]:::output
    AFT_Weights["AFT_model_weights/"]:::data

    %% Model components
    ModelManager["Model Manager"]:::model

    %% Adversarial components
    AttackGenerator["Attack Generator"]:::adversarial
    TTCModule["TTC Counterattack Module"]:::adversarial

    %% Metrics
    MetricsCollector["Evaluation & Metrics Collector"]:::orchestration

    %% Flow
    ScriptLauncher -->|"start"| ConfigParser
    ConfigParser -->|"config settings"| DatasetLoader
    ConfigParser -->|"config settings"| ModelManager
    DatasetLoader -->|"images & labels"| AttackGenerator
    AttackGenerator -->|"adversarial_images"| TTCModule
    TTCModule -->|"corrected_images"| ModelManager
    ModelManager -->|"class_scores"| MetricsCollector
    MetricsCollector -->|"metrics"| ResultSaver
    ResultSaver -->|"save results"| TTCResults

    %% Optional branch for finetuned weights
    AFT_Weights -.->|"--victim_resume"| ModelManager

    %% Click events
    click ScriptLauncher "https://github.com/sxing2/clip-test-time-counterattacks/blob/main/TTC.sh"
    click ScriptLauncher "https://github.com/sxing2/clip-test-time-counterattacks/blob/main/code/test_time_counterattack.py"
    click ConfigParser "https://github.com/sxing2/clip-test-time-counterattacks/blob/main/code/test_time_counterattack.py"
    click DatasetLoader "https://github.com/sxing2/clip-test-time-counterattacks/tree/main/code/replace/datasets/"
    click AttackGenerator "https://github.com/sxing2/clip-test-time-counterattacks/blob/main/code/attacks.py"
    click TTCModule "https://github.com/sxing2/clip-test-time-counterattacks/blob/main/code/replace/clip.py"
    click TTCModule "https://github.com/sxing2/clip-test-time-counterattacks/blob/main/code/replace/model.py"
    click TTCModule "https://github.com/sxing2/clip-test-time-counterattacks/blob/main/code/replace/simple_tokenizer.py"
    click TTCModule "https://github.com/sxing2/clip-test-time-counterattacks/blob/main/code/replace/bpe_simple_vocab_16e6.txt.gz"
    click ModelManager "https://github.com/sxing2/clip-test-time-counterattacks/blob/main/code/models/model.py"
    click ModelManager "https://github.com/sxing2/clip-test-time-counterattacks/blob/main/code/models/prompters.py"
    click ModelManager "https://github.com/sxing2/clip-test-time-counterattacks/blob/main/code/models/download_models.sh"
    click AFT_Weights "https://github.com/sxing2/clip-test-time-counterattacks/tree/main/AFT_model_weights/"
    click AFT_Weights "https://github.com/sxing2/clip-test-time-counterattacks/blob/main/download_weights.py"
    click MetricsCollector "https://github.com/sxing2/clip-test-time-counterattacks/blob/main/code/utils.py"
    click ResultSaver "https://github.com/sxing2/clip-test-time-counterattacks/blob/main/code/test_time_counterattack.py"

    %% Styles
    classDef orchestration fill:#FFA500,stroke:#333,stroke-width:1px;
    classDef data fill:#00BFFF,stroke:#333,stroke-width:1px;
    classDef model fill:#32CD32,stroke:#333,stroke-width:1px;
    classDef adversarial fill:#FF6347,stroke:#333,stroke-width:1px;
    classDef output fill:#FFD700,stroke:#333,stroke-width:1px;
```
