# bytedance-seed/bagel architecture

> Architecture diagram of the GitHub repository bytedance-seed/bagel, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bytedance-seed/bagel
- Repository: https://github.com/bytedance-seed/bagel

## Overview

#### Purpose and entry points
BAGEL is a unified multimodal foundation-model repository for visual-language understanding, text-to-image generation, image editing, and related world-modeling tasks. Primary interactive inference starts at `app.py`; programmatic/notebook inference uses `inferencer.py` and `inference.ipynb`. Training begins through `scripts/train.sh`, which drives `train/pretrain_unified_navit.py`.

#### Model assembly
The core model is BAGEL’s Mixture-of-Transformer-Experts implementation in `modeling/bagel/bagel.py`, with shared modeling support in `modeling/bagel/modeling_utils.py`. It combines a Qwen2-derived language backbone (`modeling/qwen2/modeling_qwen2.py`) with NaViT-capable vision encoders (`modeling/bagel/qwen2_navit.py`, `modeling/bagel/siglip_navit.py`). This is the principal boundary between checkpoint-loaded neural modules and application/training orchestration.

#### Multimodal token and image pipeline
Image representation is handled by the autoencoder at `modeling/autoencoder.py` and SigLIP processing/modeling modules under `modeling/siglip/processing_siglip.py` and `modeling/siglip/modeling_siglip.py`. Inputs become language and visual token groups for next-group prediction; generation/editing uses iterative denoising controls exposed in the UI/inference layer. Attention/cache optimization is isolated in `modeling/cache_utils/taylorseer.py`.

#### Inference and user-facing execution
`inferencer.py` provides the reusable model-loading and generation interface; `app.py` layers a Gradio UI on top, including quantization-oriented runtime modes. The flow is checkpoint from Hugging Face → model/processor initialization → prompt plus optional image conditioning → multimodal generation or understanding result. Pretrained artifacts are expected locally under a downloaded model directory rather than committed to the repository.

#### Training data and distributed ingestion
The `data/` package defines the training-data boundary: base metadata/contracts in `data/dataset_base.py` and `data/dataset_info.py`, modality-specific datasets in `data/t2i_dataset.py` and `data/vlm_dataset.py`, and image/video transforms in `data/transforms.py` and `data/video_utils.py`. Large-scale sources are read through parquet utilities and composed as distributed iterable/interleaved streams via `data/distributed_iterable_dataset.py` and `data/interleave_datasets/interleave_t2i_dataset.py`.

#### Distributed training
`train/pretrain_unified_navit.py` is the unified pretraining driver, consuming configured datasets and building the BAGEL model/data pipeline. `train/fsdp_utils.py` establishes the distributed/sharded-training boundary, while `train/train_utils.py` contains shared train-loop support. Dataset composition is externally parameterized through YAML, exemplified by `data/configs/example.yaml`.

#### Evaluation suites
Evaluation is partitioned by capability. VLM benchmark execution is rooted at `eval/vlm/evaluate.sh` with benchmark-specific adapters beneath `eval/vlm/eval/`. Generation and editing benchmark runners live under `eval/gen/`, including GenEval (`eval/gen/geneval/evaluation/evaluate_images.py`), GEdit (`eval/gen/gedit/gen_images_gedit.py`), and KRIS/WISE/RISE suites. Shell wrappers in `scripts/eval/` coordinate model generation and scoring workflows.

#### Runtime and external dependencies
The project targets Python 3.10 with GPU execution, PyTorch-style distributed training, and FlashAttention (`requirements.txt`). It uses Gradio for the local UI and Hugging Face Hub for checkpoint retrieval; released model weights are hosted externally. Some evaluation paths integrate external multimodal evaluators, including OpenAI-compatible scoring utilities at `eval/gen/gedit/viescore/mllm_tools/openai.py`.

Main entry points: [`modeling/bagel/bagel.py`](https://github.com/bytedance-seed/bagel/blob/main/modeling/bagel/bagel.py) (BAGEL MoT), [`inferencer.py`](https://github.com/bytedance-seed/bagel/blob/main/inferencer.py) (Inferencer), [`train/pretrain_unified_navit.py`](https://github.com/bytedance-seed/bagel/blob/main/train/pretrain_unified_navit.py) (Unified pretraining), [`data/distributed_iterable_dataset.py`](https://github.com/bytedance-seed/bagel/blob/main/data/distributed_iterable_dataset.py) (Distributed streams)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inference["Inference"]
  node_gradio_app["Gradio app<br/>UI entry point<br/>[app.py]"]
  node_inferencer["Inferencer<br/>inference API<br/>[inferencer.py]"]
  node_hf_checkpoints[("HF checkpoints<br/>external artifacts")]
end

subgraph group_model["Model stack"]
  node_bagel_model{{"BAGEL MoT<br/>core model<br/>[bagel.py]"}}
  node_model_utils["Model utilities<br/>model support<br/>[modeling_utils.py]"]
  node_qwen_backbone["Qwen2 backbone<br/>language model<br/>[modeling_qwen2.py]"]
  node_navit_encoders["NaViT encoders<br/>vision encoders<br/>[qwen2_navit.py]"]
  node_image_representation["Image representation<br/>image pipeline<br/>[autoencoder.py]"]
  node_cache_optimization["TaylorSeer cache<br/>attention optimization<br/>[taylorseer.py]"]
end

subgraph group_data_train["Data &amp; training"]
  node_data_contracts["Dataset contracts<br/>data metadata<br/>[dataset_base.py]"]
  node_modality_datasets["Modality datasets<br/>dataset loaders<br/>[t2i_dataset.py]"]
  node_distributed_streams["Distributed streams<br/>data ingestion"]
  node_dataset_yaml["Dataset YAML<br/>dataset configuration<br/>[example.yaml]"]
  node_train_script["Training launcher<br/>shell entry point<br/>[train.sh]"]
  node_pretraining_driver["Unified pretraining<br/>training driver"]
  node_fsdp["FSDP utilities<br/>distributed training<br/>[fsdp_utils.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_vlm_eval["VLM evaluation<br/>benchmark suite<br/>[evaluate.sh]"]
  node_gen_edit_eval["Generation/editing eval<br/>benchmark runners<br/>[evaluate_images.py]"]
  node_eval_wrappers["Evaluation wrappers<br/>workflow scripts"]
end

node_gradio_app -->|"uses"| node_inferencer
node_hf_checkpoints -->|"loads"| node_inferencer
node_inferencer -->|"initializes"| node_bagel_model
node_inferencer -->|"processes images"| node_image_representation
node_bagel_model -->|"language backbone"| node_qwen_backbone
node_bagel_model -->|"vision backbone"| node_navit_encoders
node_bagel_model -->|"uses"| node_model_utils
node_bagel_model -.->|"optimizes attention"| node_cache_optimization
node_image_representation -->|"visual tokens"| node_bagel_model
node_dataset_yaml -->|"configures"| node_distributed_streams
node_data_contracts -->|"defines contracts"| node_modality_datasets
node_modality_datasets -->|"composes"| node_distributed_streams
node_train_script -->|"launches"| node_pretraining_driver
node_distributed_streams -->|"feeds batches"| node_pretraining_driver
node_pretraining_driver -->|"trains"| node_bagel_model
node_pretraining_driver -->|"uses sharding"| node_fsdp
node_eval_wrappers -->|"runs"| node_vlm_eval
node_eval_wrappers -->|"runs"| node_gen_edit_eval
node_inferencer -.->|"generates outputs for"| node_eval_wrappers

click node_gradio_app "https://github.com/bytedance-seed/bagel/blob/main/app.py"
click node_inferencer "https://github.com/bytedance-seed/bagel/blob/main/inferencer.py"
click node_bagel_model "https://github.com/bytedance-seed/bagel/blob/main/modeling/bagel/bagel.py"
click node_model_utils "https://github.com/bytedance-seed/bagel/blob/main/modeling/bagel/modeling_utils.py"
click node_qwen_backbone "https://github.com/bytedance-seed/bagel/blob/main/modeling/qwen2/modeling_qwen2.py"
click node_navit_encoders "https://github.com/bytedance-seed/bagel/blob/main/modeling/bagel/qwen2_navit.py"
click node_image_representation "https://github.com/bytedance-seed/bagel/blob/main/modeling/autoencoder.py"
click node_cache_optimization "https://github.com/bytedance-seed/bagel/blob/main/modeling/cache_utils/taylorseer.py"
click node_data_contracts "https://github.com/bytedance-seed/bagel/blob/main/data/dataset_base.py"
click node_modality_datasets "https://github.com/bytedance-seed/bagel/blob/main/data/t2i_dataset.py"
click node_distributed_streams "https://github.com/bytedance-seed/bagel/blob/main/data/distributed_iterable_dataset.py"
click node_dataset_yaml "https://github.com/bytedance-seed/bagel/blob/main/data/configs/example.yaml"
click node_train_script "https://github.com/bytedance-seed/bagel/blob/main/scripts/train.sh"
click node_pretraining_driver "https://github.com/bytedance-seed/bagel/blob/main/train/pretrain_unified_navit.py"
click node_fsdp "https://github.com/bytedance-seed/bagel/blob/main/train/fsdp_utils.py"
click node_vlm_eval "https://github.com/bytedance-seed/bagel/blob/main/eval/vlm/evaluate.sh"
click node_gen_edit_eval "https://github.com/bytedance-seed/bagel/blob/main/eval/gen/geneval/evaluation/evaluate_images.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_gradio_app,node_inferencer,node_hf_checkpoints toneBlue
class node_bagel_model,node_model_utils,node_qwen_backbone,node_navit_encoders,node_image_representation,node_cache_optimization toneAmber
class node_data_contracts,node_modality_datasets,node_distributed_streams,node_dataset_yaml,node_train_script,node_pretraining_driver,node_fsdp toneMint
class node_vlm_eval,node_gen_edit_eval,node_eval_wrappers toneRose
```

## Components

### Inference

- [Gradio app](https://github.com/bytedance-seed/bagel/blob/main/app.py): UI entry point · Interactive inference with runtime quantization modes. · `app.py`
- [Inferencer](https://github.com/bytedance-seed/bagel/blob/main/inferencer.py): inference API · Loads models and runs multimodal generation. · `inferencer.py`
- **HF checkpoints**: external artifacts · Downloaded pretrained weights stored outside the repository.

### Model stack

- [BAGEL MoT](https://github.com/bytedance-seed/bagel/blob/main/modeling/bagel/bagel.py): core model · Unified mixture-of-transformer-experts model. · `modeling/bagel/bagel.py`
- [Model utilities](https://github.com/bytedance-seed/bagel/blob/main/modeling/bagel/modeling_utils.py): model support · `modeling/bagel/modeling_utils.py`
- [Qwen2 backbone](https://github.com/bytedance-seed/bagel/blob/main/modeling/qwen2/modeling_qwen2.py): language model · `modeling/qwen2/modeling_qwen2.py`
- [NaViT encoders](https://github.com/bytedance-seed/bagel/blob/main/modeling/bagel/qwen2_navit.py): vision encoders · Qwen2-NaViT and SigLIP-NaViT visual encoding. · `modeling/bagel/qwen2_navit.py`
- [Image representation](https://github.com/bytedance-seed/bagel/blob/main/modeling/autoencoder.py): image pipeline · Autoencoding and SigLIP image processing/modeling. · `modeling/autoencoder.py`
- [TaylorSeer cache](https://github.com/bytedance-seed/bagel/blob/main/modeling/cache_utils/taylorseer.py): attention optimization · `modeling/cache_utils/taylorseer.py`

### Data & training

- [Dataset contracts](https://github.com/bytedance-seed/bagel/blob/main/data/dataset_base.py): data metadata · Base dataset metadata and contracts. · `data/dataset_base.py`
- [Modality datasets](https://github.com/bytedance-seed/bagel/blob/main/data/t2i_dataset.py): dataset loaders · Text-to-image and VLM datasets. · `data/t2i_dataset.py`
- [Distributed streams](https://github.com/bytedance-seed/bagel/blob/main/data/distributed_iterable_dataset.py): data ingestion · Parquet-backed iterable and interleaved dataset streams. · `data/distributed_iterable_dataset.py`
- [Dataset YAML](https://github.com/bytedance-seed/bagel/blob/main/data/configs/example.yaml): dataset configuration · `data/configs/example.yaml`
- [Training launcher](https://github.com/bytedance-seed/bagel/blob/main/scripts/train.sh): shell entry point · `scripts/train.sh`
- [Unified pretraining](https://github.com/bytedance-seed/bagel/blob/main/train/pretrain_unified_navit.py): training driver · `train/pretrain_unified_navit.py`
- [FSDP utilities](https://github.com/bytedance-seed/bagel/blob/main/train/fsdp_utils.py): distributed training · Sharded-training boundary and support. · `train/fsdp_utils.py`

### Evaluation

- [VLM evaluation](https://github.com/bytedance-seed/bagel/blob/main/eval/vlm/evaluate.sh): benchmark suite · Capability-specific VLM benchmark adapters. · `eval/vlm/evaluate.sh`
- [Generation/editing eval](https://github.com/bytedance-seed/bagel/blob/main/eval/gen/geneval/evaluation/evaluate_images.py): benchmark runners · GenEval, GEdit, and related generation suites. · `eval/gen/geneval/evaluation/evaluate_images.py`
- **Evaluation wrappers**: workflow scripts · Coordinates generation and scoring workflows.

## Connections

- Gradio app → Inferencer: uses
- HF checkpoints → Inferencer: loads
- Inferencer → BAGEL MoT: initializes
- Inferencer → Image representation: processes images
- BAGEL MoT → Qwen2 backbone: language backbone
- BAGEL MoT → NaViT encoders: vision backbone
- BAGEL MoT → Model utilities: uses
- BAGEL MoT → TaylorSeer cache: optimizes attention
- Image representation → BAGEL MoT: visual tokens
- Dataset YAML → Distributed streams: configures
- Dataset contracts → Modality datasets: defines contracts
- Modality datasets → Distributed streams: composes
- Training launcher → Unified pretraining: launches
- Distributed streams → Unified pretraining: feeds batches
- Unified pretraining → BAGEL MoT: trains
- Unified pretraining → FSDP utilities: uses sharding
- Evaluation wrappers → VLM evaluation: runs
- Evaluation wrappers → Generation/editing eval: runs
- Inferencer → Evaluation wrappers: generates outputs for
