How roeeben/generating-band-limited-adversarial-surfaces-using-neural-networks works
Project’s Main Purpose and Type
• The README clarifies that the project is about generating adversarial examples on 3D data (meshes/point-clouds) using neural networks.
• It uses PointNet’s architecture (with modifications) both for classification and to build the adversarial network (auto-encoder based).
• The purpose is split between training a classifier (to be robust) and generating adversarial perturbations via a neural network.
• There is integration with tracking tools (e.g., Weights & Biases) and it leverages the PyTorch3D library for 3D data, rendering, and related operations.
File Structure for Architectural Components
• Top-Level Scripts and Files:
• band_limited_runner.py, classifier_trainer_main.py, model_trainer_main.py: Entry points for running attacks and for training models.
• run_config.py: Likely a configuration file that sets hyperparameters or paths, affecting training runs.
• setup.sh and requirements.txt: Environment setup and dependency specification.
• assets and saved_params: Hold visual assets (images, GIFs used in the README) and saved model checkpoints respectively.
• The src/ Directory Contains the Bulk of the Code Arranged as Follows:
a. src/adversarial:
• Contains implementations of adversarial attack algorithms (e.g., PGD, Carlini-Wagner) and supporting modules (base classes and output handlers).
• Represents the strategies or methods for generating adversarial perturbations.
b. src/dataset:
• Houses data loaders, dataset-specific files for FAUST and SHREC14, and utilities for preprocessing (e.g., downscaling).
• This is critical for ingesting the 3D mesh/point-cloud inputs and preparing them for training.
c. src/deep_adv_3d:
• Contains key components for the training of the adversarial network: custom loss functions, training loops, TensorBoard logging and utility functions.
• Implements the auto-encoder style architecture that generates perturbations.
d. src/models:
• Contains definitions of network architectures such as the modified PointNet and other model variants (e.g., deep_adv_3d_model1).
e. src/pytorch3d:
• Contains a very extensive suite of utilities and operations that extend PyTorch3D functionality.
• Covers functionalities from common tensors and types, CUDA/C++ implementations (for ball query, blending, compositing, etc.), datasets (ShapeNet, R2N2), I/O for different 3D formats, loss functions, and more.
• Serves as the backbone for all 3D processing, rendering, transformation, and differential rendering needs.
f. src/utils and src/vista:
• Utilities for mesh decimation, eigenpairs, transforms, visualization, and other helper functions.
• The vista folder, for example, is oriented toward visualizing results and plotting adversarial attacks.
Explaining the Main Components and Their Interactions
a. Data Input & Pre-Processing:
• Component: Dataset Loader (src/dataset)
• Function: Reads raw 3D data (from FAUST and SHREC14 datasets), applies downscaling or other transformations, and prepares batched input for subsequent modules.
b. Classifier (Victim Model):
• Component: PointNet (located within src/models or as a part of the adversarial training process).
c. Adversarial Generator (Attacking Network):
• Component: Auto-Encoder–Based Network (e.g., deep_adv_3d_model1 in src/models, with auxiliary codes in src/deep_adv_3d).
• Function: Generates subtle, band-limited perturbations intended to mislead the classifier. It uses modified PointNet architecture concepts.
d. Adversarial Strategies and Algorithms:
• Component: Adversarial Methods (src/adversarial)
• Function: Implements attack algorithms like PGD and Carlini-Wagner variants, which guide or refine how noise is applied.
e. 3D Processing & Rendering:
• Component: PyTorch3D Extensions (src/pytorch3d)
• Function: Provides detailed methods for mesh processing, rasterization, 3D transforms, and rendering gradients that are essential for computing differential losses and visualizing outputs.
f. Training & Experimentation Loop:
• Component: Training Loop & Loss Handling (src/deep_adv_3d/train_loop.py, src/deep_adv_3d/loss.py)
• Function: Controls experiment runs, hyperparameter sweeps (integrated with Weights & Biases), loss computation, and backpropagation.
g. Visualization & Results Output:
• Component: Vista Utilities (src/vista) and Logging (tensor_board in deep_adv_3d and utils settings).
• Function: Visualizes intermediate results including the adversarial attacks and tracked training metrics.
h. Configuration & Environment:
• Component: run_config.py, setup.sh, requirements.txt
• Function: Manage environment setup and define parameters that affect overall system behavior.
Relationships and Data Flows
• The attack’s output (adversarial perturbations) then affects the input of the classifier, closing the loop for adversarial evaluation.
• The training loop interacts with both the adversarial network and the classifier, feeding losses back to adjust weights.
• The PyTorch3D components underscore almost every step that involves 3D data conversion, rendering, and differentiable operations.
• Logs and visual outputs flow from the training components to the visualization modules (Weights & Biases/TensorBoard).
Architectural Patterns and Design Principles
• Separation of Concerns: Distinct directories are responsible for adversarial methods, data handling, network models, and low-level 3D operations.
• Modularity: Clear separation between core functionalities, allowing one to update or swap a module (for example, testing a different adversarial algorithm) without impacting others.
• Scalability: With configuration files and a clear run configuration, the architecture supports extensive hyperparameter searches and experimentation.
• Reusability: The heavy use of PyTorch3D and modular utilities (e.g., for mesh processing and visualization) ensures that components can be reused across experiments.
1. Data Input & Preprocessing (Dataset Loaders from src/dataset)
2. Classifier (PointNet module in src/models) and Adversarial Generator (Auto-encoder from src/deep_adv_3d and src/models)
3. Adversarial Methods (PGD, Carlini-Wagner implementations in src/adversarial)
4. 3D Processing & Rendering (PyTorch3D submodules from src/pytorch3d)
5. Training Loop & Loss Computation (src/deep_adv_3d/train_loop.py and loss functions)
6. Visualization & Experiment Tracking (Weights & Biases integration, TensorBoard logging, vista utils)
7. Configuration & Environment (run_config.py and setup scripts)
• Sparing interactions between modules (for example, how the adversarial algorithms alter the autoencoder’s behavior; how PyTorch3D functions are called upon for 3D operations).
• Integration points with any external frameworks/libraries.