How tencent/wemm-embedding works
WeMM-Embedding is a multimodal embedding model family; the supplied source tree exposes its MMEB-v3 evaluation pipeline rather than the documented standalone inference examples or serving wrappers. The principal visible workflow is configuring a benchmark run, loading task datasets, preparing multimodal batches, computing embeddings, and scoring retrieval results. The graph preserves documented text, image, video, visual-document, agent, MCMR, and audio evaluation coverage. Some task implementations and runtime wiring are unsampled, so uncertain dataset-to-runner links are omitted; subsystem coverage uses exact tree paths.
Main entry points: eval.py Evaluation runner · dataset_hf_path.py Dataset path maps · processor.py Multimodal processor · base_eval_dataset.py Dataset base and schema
Components 23
Benchmark orchestration
- Run arguments
mmeb_v3_eval/src/arguments.py - Evaluation runner
mmeb_v3_eval/eval.py - Benchmark scores
Task data
- Task dataset registry
mmeb_v3_eval/src/data/eval_dataset - Dataset base and schema
mmeb_v3_eval/src/data/eval_dataset/base_eval_dataset.py - Video benchmarks
mmeb_v3_eval/src/data/eval_dataset - Visual document tasks
mmeb_v3_eval/src/data/eval_dataset/vidore_dataset.py - Audio benchmarks
mmeb_v3_eval/src/data/eval_dataset/audio_cls_dataset.py - Agent and text tasks
mmeb_v3_eval/src/data/eval_dataset - MCMR tasks
mmeb_v3_eval/src/data/eval_dataset/mcmr_dataset.py - Dataset path maps
mmeb_v3_eval/src/constant/dataset_hf_path.py - Local dataset paths
mmeb_v3_eval/src/constant/dataset_hflocal_path.py - Audio instructions
mmeb_v3_eval/src/data/eval_dataset/audio_instruction_utils.py
Embedding runtime
- Embedding model
mmeb_v3_eval/src/model/model.py - Multimodal processor
mmeb_v3_eval/src/model/processor.py - Evaluation collator
mmeb_v3_eval/src/data/collator/eval_collator.py - Qwen video helpers
mmeb_v3_eval/src/model/vlm_backbone/qwen2_vl/qwen_vl_utils.py
Scoring and media
- Video and image transforms
mmeb_v3_eval/src/utils/vision_utils/vision_utils.py - Video transforms
mmeb_v3_eval/src/utils/vision_utils/video_transforms.py - Dataset loading utilities
mmeb_v3_eval/src/utils/dataset_utils.py - Ranking metrics
mmeb_v3_eval/src/utils/eval_utils/metrics.py - Distributed logging
mmeb_v3_eval/src/utils/basic_utils.py
Other components
- Evaluator
Connections 27
- Evaluator → Evaluation runner: runs evaluation
README.md - Evaluator → Run arguments: supplies settings
- Evaluation runner → Run arguments: reads settings
mmeb_v3_eval/eval.py - Evaluation runner → Embedding model: runs inference
mmeb_v3_eval/eval.py - Evaluation runner → Multimodal processor: prepares inputs
mmeb_v3_eval/eval.py - Evaluation runner → Evaluation collator: builds batches
mmeb_v3_eval/eval.py - Evaluation runner → Dataset base and schema: uses dataset base
mmeb_v3_eval/eval.py - Evaluation runner → Ranking metrics: scores rankings
mmeb_v3_eval/eval.py - Evaluation runner → Benchmark scores: writes scores
- Task dataset registry → Dataset base and schema: registers tasks
mmeb_v3_eval/src/data/eval_dataset/complex_text_retrieve.py - Video benchmarks → Dataset path maps: selects task data
mmeb_v3_eval/src/constant/dataset_hf_path.py - Visual document tasks → Dataset path maps: selects task data
mmeb_v3_eval/src/constant/dataset_hf_path.py - Audio benchmarks → Dataset path maps: selects task data
mmeb_v3_eval/src/data/eval_dataset/audio_cls_dataset.py - Audio benchmarks → Audio instructions: uses instructions
mmeb_v3_eval/src/data/eval_dataset/audio_cls_dataset.py - Agent and text tasks → Dataset base and schema: uses dataset base
mmeb_v3_eval/src/data/eval_dataset/complex_text_retrieve.py - MCMR tasks → Dataset base and schema: uses dataset base
- Dataset path maps → Local dataset paths: imports local map
mmeb_v3_eval/src/constant/dataset_hf_path.py - Audio benchmarks → Dataset loading utilities: loads datasets
mmeb_v3_eval/src/data/eval_dataset/audio_cls_dataset.py - Video benchmarks → Dataset loading utilities: loads datasets
mmeb_v3_eval/src/data/eval_dataset/mvbench_dataset.py - Visual document tasks → Dataset loading utilities: loads datasets
- Video benchmarks → Video and image transforms: uses media utilities
mmeb_v3_eval/src/data/eval_dataset/mvbench_dataset.py - Video and image transforms → Video transforms: uses transforms
mmeb_v3_eval/src/utils/vision_utils/vision_utils.py - Evaluation collator → Multimodal processor: prepares model inputs
mmeb_v3_eval/src/data/collator/eval_collator.py - Evaluation collator → Qwen video helpers: processes video
mmeb_v3_eval/src/data/collator/eval_collator.py - Embedding model → Multimodal processor: uses processor
mmeb_v3_eval/src/model/model.py - Multimodal processor → Qwen video helpers: uses video helpers
mmeb_v3_eval/src/model/processor.py - Evaluation runner → Distributed logging: logs by rank
mmeb_v3_eval/eval.py