How epfl-vilab/adversarial-prompts works
Step 1: Identify the main components
• Overall System Controller: The entry point defined in the top-level README, install scripts, and environment setup files.
• Diffusion Model & Data Generation Engine: Core routines (e.g., sd_pipeline.py, generate_adv_images*.py) that harness pretrained diffusion-based models (as evident from diffusers-base code inspiration) to produce image outputs.
• Feedback/Optimization Components:
– Adversarial Prompt Generation: Code responsible for adjusting prompts to maximize supervised model loss (e.g., diffusion_attacks.py in both classification and depth folders).
– Guided Prompt Mechanism: Components that steer the generated images toward a target distribution (e.g., using textual/image-guidance scripts and configuration files).
• Task-specific Modules:
– Classification Module: Contains configuration files (in configs/), data scripts (in data/ and clip_image_guidance.py), model definitions (in models/), and other utilities specific to classification tasks.
– Depth Estimation Module: Similar dedicated structure with its own datasets, textual inversion datasets, losses (e.g., midas_loss, masked_losses), and model files (e.g., unet.py).
• External Dependencies and Configurations:
– Config YAML files for different experiments (iwilds, waterbirds).
– Asset management (images, embeddings) to facilitate experiment visuals and recorded embeddings.
Step 2: Analyze the file structure to identify relationships
• Each sub-module includes directories for:
– Data (scripts and CSVs or datasets)
– Models (custom model definitions for each task)
– Configuration (YAML configs applying to dataset and optimization routines)
– Pipeline scripts (guiding diffusion generation and adversarial attacks)
• Shared utility scripts (e.g., utils.py) and common download or generate scripts appear for setting up data or models.
A. User/Environment Setup
• Installation and environment configuration (install.sh, requirements.txt)
• Project Setup (cloning the repo, environment activation)
B. Experiment Selector/Controller
• A high-level controller defines which task is executed (Classification or Depth).
• A decision point that directs control flow toward either the classification pipeline or depth pipeline.
C. Task-Specific Pipeline Components
For both “Classification” and “Depth” pipelines, illustrate the following:
1. Data Management Subsystem:
– Data ingestion, preprocessing, and CSV dataset definitions.
– Modules like classification/data/* and depth/datasets/*.
2. Diffusion-Based Generation Module:
– Diffusion pipelines (e.g., sd_pipeline.py, generate_adv_images*.py) that generate image data using generative diffusion models.
3. Feedback/Optimization Loop:
– Adversarial Prompts Generator: Uses supervised model feedback to maximize loss (scripts like diffusion_attacks.py).
– Guided Image Generation: Applies target distribution guidance (using text or image prompts, as seen in clip_image_guidance.py and textual_inversion_dataset*.py).
4. Model & Utility Services:
– Custom models and losses (found in classification/models/*, depth/models/unet.py, and losses/).
– Supporting utilities (common functions in utils.py).
D. External Integration and Dependencies
• Reference to external repositories (e.g., “diffusers repository”) providing inspiration or code for the adversarial feedback mechanism.
• Illustration of configuration files and asset files (from assets/ and configuration YAMLs).
– Each major subsystem (Classification and Depth) is presented in its own area/section.
– Connect shared components (like utilities and configuration management) to the appropriate modules.
– Compute modules (diffusion engines, model training)
– Storage/data modules (CSV datasets, configuration files)
– Orchestration/control elements (scripts that tie the loops together).
Because this is a research project focusing on controlled data generation using diffusion models with dual feedback:
– Emphasize the closed-loop architecture: explicitly mark the “adversarial optimization” loop (feedback from supervised loss) and the “guided” loop (target distribution-guided refinement).
– Make sure to include the separation into the two parallel experimental pipelines (Classification and Depth), showing their similarities and specific differences.
Final Guidelines for the Principal Software Engineer: