# epfl-vilab/adversarial-prompts architecture

> Architecture diagram of the GitHub repository epfl-vilab/adversarial-prompts, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/epfl-vilab/adversarial-prompts
- Repository: https://github.com/epfl-vilab/adversarial-prompts

## Overview

Step 1: Identify the main components
- Overall System Controller: The entry point defined in the top-level README, install scripts, and environment setup files.
- Diffusion Model & Data Generation Engine: Core routines (e.g., sd_pipeline.py, generate_adv_images*.py) that harness pretrained diffusion-based models (as evident from diffusers-base code inspiration) to produce image outputs.
- Feedback/Optimization Components:
– Adversarial Prompt Generation: Code responsible for adjusting prompts to maximize supervised model loss (e.g., diffusion_attacks.py in both classification and depth folders).
– Guided Prompt Mechanism: Components that steer the generated images toward a target distribution (e.g., using textual/image-guidance scripts and configuration files).
- Task-specific Modules:
– Classification Module: Contains configuration files (in configs/), data scripts (in data/ and clip_image_guidance.py), model definitions (in models/), and other utilities specific to classification tasks.
– Depth Estimation Module: Similar dedicated structure with its own datasets, textual inversion datasets, losses (e.g., midas_loss, masked_losses), and model files (e.g., unet.py).
- External Dependencies and Configurations:
– Config YAML files for different experiments (iwilds, waterbirds).
– Asset management (images, embeddings) to facilitate experiment visuals and recorded embeddings.

Step 2: Analyze the file structure to identify relationships
- Each sub-module includes directories for:
– Data (scripts and CSVs or datasets)
– Models (custom model definitions for each task)
– Configuration (YAML configs applying to dataset and optimization routines)
– Pipeline scripts (guiding diffusion generation and adversarial attacks)
- Shared utility scripts (e.g., utils.py) and common download or generate scripts appear for setting up data or models.

A. User/Environment Setup
- Installation and environment configuration (install.sh, requirements.txt)
- Project Setup (cloning the repo, environment activation)

B. Experiment Selector/Controller
- A high-level controller defines which task is executed (Classification or Depth).
- A decision point that directs control flow toward either the classification pipeline or depth pipeline.

C. Task-Specific Pipeline Components
For both “Classification” and “Depth” pipelines, illustrate the following:
1. Data Management Subsystem:
– Data ingestion, preprocessing, and CSV dataset definitions.
– Modules like classification/data/* and depth/datasets/*.
2. Diffusion-Based Generation Module:
– Diffusion pipelines (e.g., sd_pipeline.py, generate_adv_images*.py) that generate image data using generative diffusion models.
3. Feedback/Optimization Loop:
– Adversarial Prompts Generator: Uses supervised model feedback to maximize loss (scripts like diffusion_attacks.py).
– Guided Image Generation: Applies target distribution guidance (using text or image prompts, as seen in clip_image_guidance.py and textual_inversion_dataset*.py).
4. Model & Utility Services:
– Custom models and losses (found in classification/models/*, depth/models/unet.py, and losses/).
– Supporting utilities (common functions in utils.py).

D. External Integration and Dependencies
- Reference to external repositories (e.g., “diffusers repository”) providing inspiration or code for the adversarial feedback mechanism.
- Illustration of configuration files and asset files (from assets/ and configuration YAMLs).

– Each major subsystem (Classification and Depth) is presented in its own area/section.
– Connect shared components (like utilities and configuration management) to the appropriate modules.
– Compute modules (diffusion engines, model training)
– Storage/data modules (CSV datasets, configuration files)
– Orchestration/control elements (scripts that tie the loops together).

Because this is a research project focusing on controlled data generation using diffusion models with dual feedback:
– Emphasize the closed-loop architecture: explicitly mark the “adversarial optimization” loop (feedback from supervised loss) and the “guided” loop (target distribution-guided refinement).
– Make sure to include the separation into the two parallel experimental pipelines (Classification and Depth), showing their similarities and specific differences.

Final Guidelines for the Principal Software Engineer:

## Diagram (Mermaid)

```mermaid
flowchart TD

    %% User/Environment Setup
    subgraph "User/Environment Setup"
        A["Overall Controller (README)"]
        B["Overall Controller (install.sh)"]
        C["Overall Controller (requirements.txt)"]
    end

    %% Experiment Controller
    subgraph "Experiment Controller"
        D0["Task Selector"]
    end

    %% Connect Environment Setup to Controller
    A --> D0
    B --> D0
    C --> D0

    %% Classification Pipeline
    subgraph "Classification Pipeline"
        direction TB
        %% Data Management
        subgraph "Data Management"
            P["Classification Data Directory"]
            Q["Classification: iwilds.py"]
            R["Classification: waterbirds.py"]
        end

        %% Diffusion Generation
        subgraph "Diffusion Generation"
            D["Cls: SD Pipeline"]
            E["Cls: Generate iwildcam"]
            F["Cls: Generate iwilds Image Embeds"]
            G["Cls: Generate waterbirds"]
        end

        %% Feedback/Optimization
        subgraph "Feedback/Optimization"
            subgraph "Adversarial Prompt"
                J["Cls: Diffusion Attacks"]
            end
            subgraph "Guided Prompt"
                L["Cls: Clip Image Guidance"]
                M["Cls: Waterbirds Text Guidance"]
            end
        end

        %% Models & Configuration
        subgraph "Models & Configuration"
            S["Classification Configs"]
            T["Classification Models"]
            U["Classification Utilities"]
        end
    end

    %% Depth Pipeline
    subgraph "Depth Pipeline"
        direction TB
        %% Data Management
        subgraph "Data Management"
            V["Depth Datasets Directory"]
        end

        %% Diffusion Generation
        subgraph "Diffusion Generation"
            H["Depth: Generate adv images"]
            I["Depth: Generate adv images omnidata"]
        end

        %% Feedback/Optimization
        subgraph "Feedback/Optimization"
            subgraph "Adversarial Prompt"
                K["Depth: Diffusion Attacks Image Guidance"]
            end
            subgraph "Guided Prompt"
                N["Depth: Textual Inversion Dataset"]
                O["Depth: Textual Inversion Dataset Omnidata"]
            end
        end

        %% Models & Other Components
        subgraph "Models & Other"
            W["Depth: UNet Model"]
            X["Depth Losses"]
            Y["Depth Additional Components"]
            Z["Depth Utilities"]
        end
    end

    %% External Integration
    subgraph "External Integration"
        AA["Assets & External Configs"]
    end

    %% Connections for Classification Pipeline
    D0 --> P
    P --> D
    D --> E
    E --> F
    F --> G

    %% Feedback loops in Classification Pipeline
    G --> J
    J --> D
    G --> L
    L --> D
    D --> M
    M --> D

    %% Connect Diffusion Generation to Models & Configuration
    G --> T
    T --> U
    U --> S
    S --> AA

    %% Connections for Depth Pipeline
    D0 --> V
    V --> H
    H --> I

    %% Feedback loops in Depth Pipeline
    I --> K
    K --> H
    I --> N
    N --> H
    I --> O
    O --> H

    %% Connect Diffusion Generation to Models & Other
    I --> W
    W --> X
    W --> Y
    W --> Z
    Z --> AA

    %% Styles
    classDef envfill fill:#ccffcc,stroke:#333,stroke-width:2px;
    classDef controller fill:#ffeb99,stroke:#333,stroke-width:2px;
    classDef classification fill:#aed581,stroke:#333,stroke-width:2px;
    classDef diffusion fill:#81d4fa,stroke:#333,stroke-width:2px;
    classDef feedback fill:#ffab91,stroke:#333,stroke-width:2px;
    classDef model fill:#ce93d8,stroke:#333,stroke-width:2px;
    classDef external fill:#b0bec5,stroke:#333,stroke-width:2px;

    %% Assign styles for User/Environment Setup nodes
    class A,B,C envfill;

    %% Experiment Controller node style
    class D0 controller;

    %% Classification Pipeline Nodes
    class P,Q,R classification;
    class D,E,F,G diffusion;
    class J,L,M feedback;
    class S,T,U model;

    %% Depth Pipeline Nodes
    class V classification;
    class H,I diffusion;
    class K,N,O feedback;
    class W,X,Y,Z model;

    %% External Integration node style
    class AA external;

    %% Click Events for Overall System Controller
    click A "https://github.com/epfl-vilab/adversarial-prompts/blob/main/README.md"
    click B "https://github.com/epfl-vilab/adversarial-prompts/blob/main/install.sh"
    click C "https://github.com/epfl-vilab/adversarial-prompts/blob/main/requirements.txt"

    %% Click Events for Diffusion Model & Data Generation Engine - Classification
    click D "https://github.com/epfl-vilab/adversarial-prompts/blob/main/classification/sd_pipeline.py"
    click E "https://github.com/epfl-vilab/adversarial-prompts/blob/main/classification/generate_iwildcam.py"
    click F "https://github.com/epfl-vilab/adversarial-prompts/blob/main/classification/generate_iwilds_image_embeds.py"
    click G "https://github.com/epfl-vilab/adversarial-prompts/blob/main/classification/generate_waterbirds.py"

    %% Click Events for Diffusion Model & Data Generation Engine - Depth
    click H "https://github.com/epfl-vilab/adversarial-prompts/blob/main/depth/generate_adv_images.py"
    click I "https://github.com/epfl-vilab/adversarial-prompts/blob/main/depth/generate_adv_images_omnidata.py"

    %% Click Events for Adversarial Prompt Generation
    click J "https://github.com/epfl-vilab/adversarial-prompts/blob/main/classification/diffusion_attacks.py"
    click K "https://github.com/epfl-vilab/adversarial-prompts/blob/main/depth/diffusion_attacks_image_guidance.py"

    %% Click Events for Guided Prompt Mechanism
    click L "https://github.com/epfl-vilab/adversarial-prompts/blob/main/classification/clip_image_guidance.py"
    click M "https://github.com/epfl-vilab/adversarial-prompts/blob/main/classification/waterbirds_text_guidance.pt"
    click N "https://github.com/epfl-vilab/adversarial-prompts/blob/main/depth/datasets/textual_inversion_dataset.py"
    click O "https://github.com/epfl-vilab/adversarial-prompts/blob/main/depth/datasets/textual_inversion_dataset_omnidata.py"

    %% Click Events for Classification Module
    click P "https://github.com/epfl-vilab/adversarial-prompts/tree/main/classification/data/"
    click Q "https://github.com/epfl-vilab/adversarial-prompts/blob/main/classification/data/iwilds.py"
    click R "https://github.com/epfl-vilab/adversarial-prompts/blob/main/classification/data/waterbirds.py"
    click S "https://github.com/epfl-vilab/adversarial-prompts/tree/main/classification/configs/"
    click T "https://github.com/epfl-vilab/adversarial-prompts/tree/main/classification/models/"
    click U "https://github.com/epfl-vilab/adversarial-prompts/blob/main/classification/utils.py"

    %% Click Events for Depth Estimation Module
    click V "https://github.com/epfl-vilab/adversarial-prompts/tree/main/depth/datasets/"
    click W "https://github.com/epfl-vilab/adversarial-prompts/blob/main/depth/models/unet.py"
    click X "https://github.com/epfl-vilab/adversarial-prompts/tree/main/depth/losses/"
    click Y "https://github.com/epfl-vilab/adversarial-prompts/tree/main/depth/midas/"
    click Z "https://github.com/epfl-vilab/adversarial-prompts/blob/main/depth/utils.py"

    %% Click Event for Asset and External Configurations
    click AA "https://github.com/epfl-vilab/adversarial-prompts/tree/main/assets"
```
