How laitifranz/prompt2guard works
Main Components
• Data & Configuration:
• Datasets and Data Preparation (see the README’s “Preparing Dataset” section).
• Configuration files in the “configs” directory (e.g., cddb_training.json and cddb_inference.json) that define experiment parameters.
• Model Architecture:
• The backbone VLM (CLIP) found in the “src/models/clip” folder (includes clip.py, model.py, prompt_learner.py, and a simple tokenizer).
• Additional model components like “src/models/slinet.py” for any supplementary modules.
• Methodology / Prompt Learning:
• The “src/methods” folder that contains prompt2guard.py (where the core method is implemented and the text-prompt conditioning is handled).
• Training & Evaluation Pipeline:
• Entry points: “src/train.py” for training and “src/eval.py” for evaluation.
• Supporting modules such as “src/trainer.py” and “src/utils” for data management, learning rate scheduling, zero-shot classification, and other utilities.
• Checkpoints and Pretrained Weights:
• The “checkpoints” directory holds pretrained weights, and this integrates into both training and evaluation pipelines.
Relationships and Data Flow
• Input Data Flow:
• Raw images are provided from external datasets (e.g., CDDB) and managed via “src/utils/data.py” and “src/utils/data_manager.py”.
• Classes and associated probabilities are loaded from “src/utils/classes.pkl”, produced through or used by “src/utils/zeroshot_classprediction.py”.
• Core Model Pipeline:
• Processed inputs are forwarded through the CLIP backbone (from the “src/models/clip” module).
Input Images -> Data Preparation -> CLIP backbone (with tokenization and prompt learning) -> Prompt2Guard module.
• The ensembling (combining predictions at various stages, as mentioned in the README configs) is applied after cosine similarity computation.
• Training and Monitoring:
• “src/train.py” orchestrates the training loop, calling on “src/trainer.py” and leveraging utilities for scheduling and checkpointing.
• Integration with wandb (Weights & Biases) for tracking metrics.
• Evaluation Pipeline:
• “src/eval.py” leverages the same model architecture with applied checkpoints to evaluate on multiple scenarios (cddb_hard and/or OOD datasets).
Architectural Patterns and Design Principles to Highlight
• Modularity: Separation of concerns is clear – models, methods, utilities, and configuration are isolated into separate directories.
• Config-driven Behavior: The use of JSON config files for training and inference enables easy reproducing and switching between scenarios.
• Extensibility: The clear separation of “prompt2guard” logic, coupled with the modular design of data handling and model definitions, supports future extensions or changes (e.g., using a different prompt optimizer).
• Efficiency: The design emphasizes prompt-conditioning to reduce computation (by avoiding multiple forward passes), which is a critical design principle in the method.
Final Advice for the Principal Software Engineer
• Don’t hesitate to break down complex nodes (like the model layer) into sub-components (e.g., token embedding, cosine similarity, ensembling logic) to capture the method’s multi-stage architecture.