# yiyanxu/pigeon architecture

> Architecture diagram of the GitHub repository yiyanxu/pigeon, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yiyanxu/pigeon
- Repository: https://github.com/yiyanxu/pigeon

## Overview

Pigeon personalizes image generation from a user’s interaction history and a target instruction: it prepares multimodal inputs, masks history/reference image tokens, produces personalized tokens, and decodes them into an image. Its documented lifecycle also includes supervised alignment, preference-pair construction and DPO alignment, plus image-quality and personalization evaluation. The graph reflects README-described capabilities where implementation excerpts were unavailable; uncertain internal wiring is omitted.

Main entry points: [`Pigeon/models/lavit_for_pigeon.py`](https://github.com/yiyanxu/pigeon/blob/main/Pigeon/models/lavit_for_pigeon.py) (Personalized multimodal model), [`Pigeon/inference.py`](https://github.com/yiyanxu/pigeon/blob/main/Pigeon/inference.py) (Image generation), [`Pigeon/finetune_dpo.py`](https://github.com/yiyanxu/pigeon/blob/main/Pigeon/finetune_dpo.py) (DPO alignment), [`Pigeon/data_utils.py`](https://github.com/yiyanxu/pigeon/blob/main/Pigeon/data_utils.py) (Dataset preparation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data Preparation"]
  node_data_utils["Dataset preparation<br/>[data_utils.py]"]
  node_data_utils_dpo["DPO data preparation<br/>[data_utils_dpo.py]"]
end

subgraph group_generation["Personalized Generation"]
  node_masking["Token mask generation"]
  node_pigeon_model["Personalized multimodal model"]
  node_visual_encoder["Visual encoder"]
  node_visual_tokenizer["Visual tokenizer"]
  node_decoder["Image token decoder"]
  node_inference["Image generation<br/>[inference.py]"]
end

subgraph group_alignment["Model Alignment"]
  node_sft["SFT alignment<br/>[finetune_sft.py]"]
  node_sft_trainer["Pigeon trainer"]
  node_dpo["DPO alignment<br/>[finetune_dpo.py]"]
  node_dpo_trainer["DPO trainer<br/>[dpo_trainer.py]"]
  node_preferences["Preference pair selection"]
end

subgraph group_evaluation["Evaluation"]
  node_score_calculation["History/reference scores<br/>[cal_scores.py]"]
  node_eval_utils["Evaluation utilities<br/>[eval_utils.py]"]
  node_evaluation["Image evaluation<br/>[evaluate_pigeon.py]"]
end

node_user(("User history + instruction"))
node_datasets[("Interaction datasets")]
node_pretrained[("Pretrained models")]
node_generated["Personalized images"]
node_results["Evaluation metrics"]

node_datasets -->|"prepares samples"| node_data_utils
node_data_utils -->|"provides training data"| node_sft
node_data_utils -->|"provides evaluation data"| node_inference
node_datasets -->|"prepares preferences"| node_data_utils_dpo
node_data_utils_dpo -->|"provides DPO data"| node_dpo
node_user -->|"supplies context"| node_inference
node_pretrained -.->|"initializes model"| node_pigeon_model
node_sft -->|"configures model"| node_pigeon_model
node_sft -->|"trains with"| node_sft_trainer
node_sft_trainer -->|"optimizes"| node_pigeon_model
node_inference -->|"runs inference"| node_pigeon_model
node_pigeon_model -->|"generates masks"| node_masking
node_pigeon_model -->|"encodes images"| node_visual_encoder
node_pigeon_model -->|"tokenizes images"| node_visual_tokenizer
node_pigeon_model -->|"decodes image tokens"| node_decoder
node_inference -->|"saves images"| node_generated
node_generated -->|"scores against history"| node_score_calculation
node_score_calculation -->|"uses utilities"| node_eval_utils
node_score_calculation -->|"provides scores"| node_preferences
node_preferences -->|"constructs preference data"| node_dpo
node_dpo -->|"loads SFT model"| node_pigeon_model
node_dpo -->|"trains with"| node_dpo_trainer
node_dpo_trainer -->|"optimizes preference"| node_pigeon_model
node_generated -->|"evaluates images"| node_evaluation
node_evaluation -->|"uses utilities"| node_eval_utils
node_evaluation -->|"reports metrics"| node_results

click node_data_utils "https://github.com/yiyanxu/pigeon/blob/main/Pigeon/data_utils.py"
click node_data_utils_dpo "https://github.com/yiyanxu/pigeon/blob/main/Pigeon/data_utils_dpo.py"
click node_masking "https://github.com/yiyanxu/pigeon/blob/main/Pigeon/models/mutual_mask_generator.py"
click node_pigeon_model "https://github.com/yiyanxu/pigeon/blob/main/Pigeon/models/lavit_for_pigeon.py"
click node_visual_encoder "https://github.com/yiyanxu/pigeon/blob/main/Pigeon/models/modeling_visual_encoder.py"
click node_visual_tokenizer "https://github.com/yiyanxu/pigeon/blob/main/Pigeon/models/modeling_visual_tokenzier.py"
click node_decoder "https://github.com/yiyanxu/pigeon/blob/main/Pigeon/models/modeling_decoder.py"
click node_inference "https://github.com/yiyanxu/pigeon/blob/main/Pigeon/inference.py"
click node_sft "https://github.com/yiyanxu/pigeon/blob/main/Pigeon/finetune_sft.py"
click node_sft_trainer "https://github.com/yiyanxu/pigeon/blob/main/Pigeon/models/trainer/trainer_for_pigeon.py"
click node_dpo "https://github.com/yiyanxu/pigeon/blob/main/Pigeon/finetune_dpo.py"
click node_dpo_trainer "https://github.com/yiyanxu/pigeon/blob/main/Pigeon/models/trainer/dpo_trainer.py"
click node_preferences "https://github.com/yiyanxu/pigeon/blob/main/Evaluation/select_scores_dpo.ipynb"
click node_score_calculation "https://github.com/yiyanxu/pigeon/blob/main/Evaluation/cal_scores.py"
click node_eval_utils "https://github.com/yiyanxu/pigeon/blob/main/Evaluation/eval_utils.py"
click node_evaluation "https://github.com/yiyanxu/pigeon/blob/main/Evaluation/evaluate_pigeon.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_data_utils,node_data_utils_dpo,node_user toneBlue
class node_masking,node_pigeon_model,node_visual_encoder,node_visual_tokenizer,node_decoder,node_inference,node_datasets,node_pretrained toneAmber
class node_sft,node_sft_trainer,node_dpo,node_dpo_trainer,node_preferences toneMint
class node_score_calculation,node_eval_utils,node_evaluation toneRose
class node_generated,node_results toneIndigo
```

## Components

### Data Preparation

- [Dataset preparation](https://github.com/yiyanxu/pigeon/blob/main/Pigeon/data_utils.py): `Pigeon/data_utils.py`
- [DPO data preparation](https://github.com/yiyanxu/pigeon/blob/main/Pigeon/data_utils_dpo.py): `Pigeon/data_utils_dpo.py`

### Personalized Generation

- [Token mask generation](https://github.com/yiyanxu/pigeon/blob/main/Pigeon/models/mutual_mask_generator.py): `Pigeon/models/mutual_mask_generator.py`
- [Personalized multimodal model](https://github.com/yiyanxu/pigeon/blob/main/Pigeon/models/lavit_for_pigeon.py): `Pigeon/models/lavit_for_pigeon.py`
- [Visual encoder](https://github.com/yiyanxu/pigeon/blob/main/Pigeon/models/modeling_visual_encoder.py): `Pigeon/models/modeling_visual_encoder.py`
- [Visual tokenizer](https://github.com/yiyanxu/pigeon/blob/main/Pigeon/models/modeling_visual_tokenzier.py): `Pigeon/models/modeling_visual_tokenzier.py`
- [Image token decoder](https://github.com/yiyanxu/pigeon/blob/main/Pigeon/models/modeling_decoder.py): `Pigeon/models/modeling_decoder.py`
- [Image generation](https://github.com/yiyanxu/pigeon/blob/main/Pigeon/inference.py): `Pigeon/inference.py`

### Model Alignment

- [SFT alignment](https://github.com/yiyanxu/pigeon/blob/main/Pigeon/finetune_sft.py): `Pigeon/finetune_sft.py`
- [Pigeon trainer](https://github.com/yiyanxu/pigeon/blob/main/Pigeon/models/trainer/trainer_for_pigeon.py): `Pigeon/models/trainer/trainer_for_pigeon.py`
- [DPO alignment](https://github.com/yiyanxu/pigeon/blob/main/Pigeon/finetune_dpo.py): `Pigeon/finetune_dpo.py`
- [DPO trainer](https://github.com/yiyanxu/pigeon/blob/main/Pigeon/models/trainer/dpo_trainer.py): `Pigeon/models/trainer/dpo_trainer.py`
- [Preference pair selection](https://github.com/yiyanxu/pigeon/blob/main/Evaluation/select_scores_dpo.ipynb): `Evaluation/select_scores_dpo.ipynb`

### Evaluation

- [History/reference scores](https://github.com/yiyanxu/pigeon/blob/main/Evaluation/cal_scores.py): `Evaluation/cal_scores.py`
- [Evaluation utilities](https://github.com/yiyanxu/pigeon/blob/main/Evaluation/eval_utils.py): `Evaluation/eval_utils.py`
- [Image evaluation](https://github.com/yiyanxu/pigeon/blob/main/Evaluation/evaluate_pigeon.py): `Evaluation/evaluate_pigeon.py`

### Other components

- **User history + instruction**
- **Interaction datasets**
- **Pretrained models**
- **Personalized images**
- **Evaluation metrics**

## Connections

- Interaction datasets → Dataset preparation: prepares samples
- Dataset preparation → SFT alignment: provides training data
- Dataset preparation → Image generation: provides evaluation data
- Interaction datasets → DPO data preparation: prepares preferences
- DPO data preparation → DPO alignment: provides DPO data
- User history + instruction → Image generation: supplies context
- Pretrained models → Personalized multimodal model: initializes model
- SFT alignment → Personalized multimodal model: configures model
- SFT alignment → Pigeon trainer: trains with
- Pigeon trainer → Personalized multimodal model: optimizes
- Image generation → Personalized multimodal model: runs inference
- Personalized multimodal model → Token mask generation: generates masks
- Personalized multimodal model → Visual encoder: encodes images
- Personalized multimodal model → Visual tokenizer: tokenizes images
- Personalized multimodal model → Image token decoder: decodes image tokens
- Image generation → Personalized images: saves images
- Personalized images → History/reference scores: scores against history
- History/reference scores → Evaluation utilities: uses utilities
- History/reference scores → Preference pair selection: provides scores
- Preference pair selection → DPO alignment: constructs preference data
- DPO alignment → Personalized multimodal model: loads SFT model
- DPO alignment → DPO trainer: trains with
- DPO trainer → Personalized multimodal model: optimizes preference
- Personalized images → Image evaluation: evaluates images
- Image evaluation → Evaluation utilities: uses utilities
- Image evaluation → Evaluation metrics: reports metrics