How dbsxodud-11/pag works
Project type and purpose
• This is an ML research/training pipeline (“PAG”) for learning to sample effective and diverse text‐to‐image prompts.
• It orchestrates data preparation, model training (on GPT-2 with specialized objectives), aesthetic scoring (via Stable Diffusion + CLIP models), and final evaluation.
File structure
• Top level:
• main.py – CLI entrypoint for training
• eval.py – CLI entrypoint for evaluation
• dataset.py – data loading & preprocessing
• gfn_trainer.py – core training loop & objective functions
• utils.py – shared helper functions
• save/ – holds pretrained and newly trained GPT-2 SFT checkpoints
• prompts/ – JSONL prompt datasets + convert_format.ipynb
• aesthetic/ – submodule containing pretrained aesthetic predictors, inference scripts, and training code
• requirements.txt – Python dependencies
• Patterns:
• Clear separation of entrypoints (main vs. eval), data-layer (dataset.py + prompts), model-layer (save/, aesthetic/), and training logic (gfn_trainer.py).
Key insights from README
• Uses GPT-2 (“lm_name”) and Stable Diffusion (“sd_name”) with aesthetic scoring (“aes”).
• Workflow: install deps → download pretrained SFT and prompts → run convert_format → train with main.py → evaluate with eval.py.
• External resources: HuggingFace CLIP, Stable-Diffusion checkpoints, Microsoft LMOps Promptist data.
Main components & interactions (
a. Data Ingestion Layer
• prompts/* (raw JSONL) → convert_format.ipynb → standardized dataset → dataset.py
b. Training Orchestrator
• main.py
• invokes dataset.py to load prompts
• constructs model:
• GPT-2 SFT (pretrained in save/gpt2-sft-position-final)
• Stable Diffusion (CompVis/stable-diffusion-v1-4 via API)
• Aesthetic predictor (aesthetic/inference.py loads .pth models)
• gfn_trainer.py implements: flow reactivation, reward decomposition, reweighted training
• utils.py for logging, checkpointing
• output: new SFT checkpoint → save/pag
c. Evaluation Orchestrator
• eval.py loads save/pag checkpoint + prompts for evaluation
• runs generation + aesthetic scoring to produce metrics
d. External Services & Dependencies
• HuggingFace Transformers & Diffusers
• OpenAI CLIP (installed via git+https://github.com/openai/CLIP.git)
• Pretrained weights (placed under save/ and aesthetic/)
Architectural patterns & design principles
• Modular: distinct layers for data, model, training logic, evaluation
• CLI-driven orchestration (main.py / eval.py) for reproducibility
• Pluggable scoring modules (aesthetic/)
• Separation of concerns: dataset vs. trainer vs. inference vs. evaluation