How yiyanxu/pigeon works
Pigeon personalizes image generation from a user’s interaction history and a target instruction: it prepares multimodal inputs, masks history/reference image tokens, produces personalized tokens, and decodes them into an image. Its documented lifecycle also includes supervised alignment, preference-pair construction and DPO alignment, plus image-quality and personalization evaluation. The graph reflects README-described capabilities where implementation excerpts were unavailable; uncertain internal wiring is omitted.
Main entry points: lavit_for_pigeon.py Personalized multimodal model · inference.py Image generation · finetune_dpo.py DPO alignment · data_utils.py Dataset preparation
Components 21
Data Preparation
- Dataset preparation
Pigeon/data_utils.py - DPO data preparation
Pigeon/data_utils_dpo.py
Personalized Generation
- Token mask generation
Pigeon/models/mutual_mask_generator.py - Personalized multimodal model
Pigeon/models/lavit_for_pigeon.py - Visual encoder
Pigeon/models/modeling_visual_encoder.py - Visual tokenizer
Pigeon/models/modeling_visual_tokenzier.py - Image token decoder
Pigeon/models/modeling_decoder.py - Image generation
Pigeon/inference.py
Model Alignment
- SFT alignment
Pigeon/finetune_sft.py - Pigeon trainer
Pigeon/models/trainer/trainer_for_pigeon.py - DPO alignment
Pigeon/finetune_dpo.py - DPO trainer
Pigeon/models/trainer/dpo_trainer.py - Preference pair selection
Evaluation/select_scores_dpo.ipynb
Evaluation
- History/reference scores
Evaluation/cal_scores.py - Evaluation utilities
Evaluation/eval_utils.py - Image evaluation
Evaluation/evaluate_pigeon.py
Other components
- User history + instruction
- Interaction datasets
- Pretrained models
- Personalized images
- Evaluation metrics
Connections 26
- Interaction datasets → Dataset preparation: prepares samples
- Dataset preparation → SFT alignment: provides training data
- Dataset preparation → Image generation: provides evaluation data
- Interaction datasets → DPO data preparation: prepares preferences
- DPO data preparation → DPO alignment: provides DPO data
- User history + instruction → Image generation: supplies context
- Pretrained models → Personalized multimodal model: initializes model
- SFT alignment → Personalized multimodal model: configures model
- SFT alignment → Pigeon trainer: trains with
- Pigeon trainer → Personalized multimodal model: optimizes
- Image generation → Personalized multimodal model: runs inference
- Personalized multimodal model → Token mask generation: generates masks
- Personalized multimodal model → Visual encoder: encodes images
- Personalized multimodal model → Visual tokenizer: tokenizes images
- Personalized multimodal model → Image token decoder: decodes image tokens
- Image generation → Personalized images: saves images
- Personalized images → History/reference scores: scores against history
- History/reference scores → Evaluation utilities: uses utilities
- History/reference scores → Preference pair selection: provides scores
- Preference pair selection → DPO alignment: constructs preference data
- DPO alignment → Personalized multimodal model: loads SFT model
- DPO alignment → DPO trainer: trains with
- DPO trainer → Personalized multimodal model: optimizes preference
- Personalized images → Image evaluation: evaluates images
- Image evaluation → Evaluation utilities: uses utilities
- Image evaluation → Evaluation metrics: reports metrics