How bytedance-seed/bagel works
Purpose and entry points
BAGEL is a unified multimodal foundation-model repository for visual-language understanding, text-to-image generation, image editing, and related world-modeling tasks. Primary interactive inference starts at app.py; programmatic/notebook inference uses inferencer.py and inference.ipynb. Training begins through scripts/train.sh, which drives train/pretrain_unified_navit.py.
Model assembly
The core model is BAGEL’s Mixture-of-Transformer-Experts implementation in modeling/bagel/bagel.py, with shared modeling support in modeling/bagel/modeling_utils.py. It combines a Qwen2-derived language backbone (modeling/qwen2/modeling_qwen2.py) with NaViT-capable vision encoders (modeling/bagel/qwen2_navit.py, modeling/bagel/siglip_navit.py). This is the principal boundary between checkpoint-loaded neural modules and application/training orchestration.
Multimodal token and image pipeline
Image representation is handled by the autoencoder at modeling/autoencoder.py and SigLIP processing/modeling modules under modeling/siglip/processing_siglip.py and modeling/siglip/modeling_siglip.py. Inputs become language and visual token groups for next-group prediction; generation/editing uses iterative denoising controls exposed in the UI/inference layer. Attention/cache optimization is isolated in modeling/cache_utils/taylorseer.py.
Inference and user-facing execution
inferencer.py provides the reusable model-loading and generation interface; app.py layers a Gradio UI on top, including quantization-oriented runtime modes. The flow is checkpoint from Hugging Face → model/processor initialization → prompt plus optional image conditioning → multimodal generation or understanding result. Pretrained artifacts are expected locally under a downloaded model directory rather than committed to the repository.
Training data and distributed ingestion
The data/ package defines the training-data boundary: base metadata/contracts in data/dataset_base.py and data/dataset_info.py, modality-specific datasets in data/t2i_dataset.py and data/vlm_dataset.py, and image/video transforms in data/transforms.py and data/video_utils.py. Large-scale sources are read through parquet utilities and composed as distributed iterable/interleaved streams via data/distributed_iterable_dataset.py and data/interleave_datasets/interleave_t2i_dataset.py.
Distributed training
train/pretrain_unified_navit.py is the unified pretraining driver, consuming configured datasets and building the BAGEL model/data pipeline. train/fsdp_utils.py establishes the distributed/sharded-training boundary, while train/train_utils.py contains shared train-loop support. Dataset composition is externally parameterized through YAML, exemplified by data/configs/example.yaml.
Evaluation suites
Evaluation is partitioned by capability. VLM benchmark execution is rooted at eval/vlm/evaluate.sh with benchmark-specific adapters beneath eval/vlm/eval/. Generation and editing benchmark runners live under eval/gen/, including GenEval (eval/gen/geneval/evaluation/evaluate_images.py), GEdit (eval/gen/gedit/gen_images_gedit.py), and KRIS/WISE/RISE suites. Shell wrappers in scripts/eval/ coordinate model generation and scoring workflows.
Runtime and external dependencies
The project targets Python 3.10 with GPU execution, PyTorch-style distributed training, and FlashAttention (requirements.txt). It uses Gradio for the local UI and Hugging Face Hub for checkpoint retrieval; released model weights are hosted externally. Some evaluation paths integrate external multimodal evaluators, including OpenAI-compatible scoring utilities at eval/gen/gedit/viescore/mllm_tools/openai.py.
Main entry points: bagel.py BAGEL MoT · inferencer.py Inferencer · pretrain_unified_navit.py Unified pretraining · distributed_iterable_dataset.py Distributed streams
Components 19
Inference
- Gradio app · Interactive inference with runtime quantization modes.
app.py - Inferencer · Loads models and runs multimodal generation.
inferencer.py - HF checkpoints · Downloaded pretrained weights stored outside the repository.
Model stack
- BAGEL MoT · Unified mixture-of-transformer-experts model.
modeling/bagel/bagel.py - Model utilities
modeling/bagel/modeling_utils.py - Qwen2 backbone
modeling/qwen2/modeling_qwen2.py - NaViT encoders · Qwen2-NaViT and SigLIP-NaViT visual encoding.
modeling/bagel/qwen2_navit.py - Image representation · Autoencoding and SigLIP image processing/modeling.
modeling/autoencoder.py - TaylorSeer cache
modeling/cache_utils/taylorseer.py
Data & training
- Dataset contracts · Base dataset metadata and contracts.
data/dataset_base.py - Modality datasets · Text-to-image and VLM datasets.
data/t2i_dataset.py - Distributed streams · Parquet-backed iterable and interleaved dataset streams.
data/distributed_iterable_dataset.py - Dataset YAML
data/configs/example.yaml - Training launcher
scripts/train.sh - Unified pretraining
train/pretrain_unified_navit.py - FSDP utilities · Sharded-training boundary and support.
train/fsdp_utils.py
Evaluation
- VLM evaluation · Capability-specific VLM benchmark adapters.
eval/vlm/evaluate.sh - Generation/editing eval · GenEval, GEdit, and related generation suites.
eval/gen/geneval/evaluation/evaluate_images.py - Evaluation wrappers · Coordinates generation and scoring workflows.
Connections 19
- Gradio app → Inferencer: uses
- HF checkpoints → Inferencer: loads
- Inferencer → BAGEL MoT: initializes
- Inferencer → Image representation: processes images
- BAGEL MoT → Qwen2 backbone: language backbone
- BAGEL MoT → NaViT encoders: vision backbone
- BAGEL MoT → Model utilities: uses
- BAGEL MoT → TaylorSeer cache: optimizes attention
- Image representation → BAGEL MoT: visual tokens
- Dataset YAML → Distributed streams: configures
- Dataset contracts → Modality datasets: defines contracts
- Modality datasets → Distributed streams: composes
- Training launcher → Unified pretraining: launches
- Distributed streams → Unified pretraining: feeds batches
- Unified pretraining → BAGEL MoT: trains
- Unified pretraining → FSDP utilities: uses sharding
- Evaluation wrappers → VLM evaluation: runs
- Evaluation wrappers → Generation/editing eval: runs
- Inferencer → Evaluation wrappers: generates outputs for