# tencent/wemm-embedding architecture

> Architecture diagram of the GitHub repository tencent/wemm-embedding, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tencent/wemm-embedding
- Repository: https://github.com/tencent/wemm-embedding

## Overview

WeMM-Embedding is a multimodal embedding model family; the supplied source tree exposes its MMEB-v3 evaluation pipeline rather than the documented standalone inference examples or serving wrappers. The principal visible workflow is configuring a benchmark run, loading task datasets, preparing multimodal batches, computing embeddings, and scoring retrieval results. The graph preserves documented text, image, video, visual-document, agent, MCMR, and audio evaluation coverage. Some task implementations and runtime wiring are unsampled, so uncertain dataset-to-runner links are omitted; subsystem coverage uses exact tree paths.

Main entry points: [`mmeb_v3_eval/eval.py`](https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/eval.py) (Evaluation runner), [`mmeb_v3_eval/src/constant/dataset_hf_path.py`](https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/constant/dataset_hf_path.py) (Dataset path maps), [`mmeb_v3_eval/src/model/processor.py`](https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/model/processor.py) (Multimodal processor), [`mmeb_v3_eval/src/data/eval_dataset/base_eval_dataset.py`](https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/data/eval_dataset/base_eval_dataset.py) (Dataset base and schema)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_benchmark["Benchmark orchestration"]
  node_config["Run arguments<br/>[arguments.py]"]
  node_runner["Evaluation runner<br/>[eval.py]"]
  node_results["Benchmark scores"]
end

subgraph group_datasets["Task data"]
  node_task_registry["Task dataset registry"]
  node_base_dataset["Dataset base and schema"]
  node_video_tasks["Video benchmarks"]
  node_visual_docs["Visual document tasks<br/>[vidore_dataset.py]"]
  node_audio_tasks["Audio benchmarks"]
  node_agent_tasks["Agent and text tasks"]
  node_mcmr_tasks["MCMR tasks<br/>[mcmr_dataset.py]"]
  node_dataset_paths["Dataset path maps<br/>[dataset_hf_path.py]"]
  node_local_paths["Local dataset paths"]
  node_audio_prompts["Audio instructions"]
end

subgraph group_embedding["Embedding runtime"]
  node_model["Embedding model<br/>[model.py]"]
  node_processor["Multimodal processor<br/>[processor.py]"]
  node_batcher["Evaluation collator<br/>[eval_collator.py]"]
  node_qwen_video["Qwen video helpers<br/>[qwen_vl_utils.py]"]
end

subgraph group_support["Scoring and media"]
  node_vision["Video and image transforms<br/>[vision_utils.py]"]
  node_video_transforms["Video transforms"]
  node_dataset_io["Dataset loading utilities<br/>[dataset_utils.py]"]
  node_rank_metrics["Ranking metrics<br/>[metrics.py]"]
  node_rank_utils["Distributed logging<br/>[basic_utils.py]"]
end

node_user(("Evaluator"))

node_user -->|"runs evaluation"| node_runner
node_user -.->|"supplies settings"| node_config
node_runner -->|"reads settings"| node_config
node_runner -->|"runs inference"| node_model
node_runner -->|"prepares inputs"| node_processor
node_runner -->|"builds batches"| node_batcher
node_runner -->|"uses dataset base"| node_base_dataset
node_runner -->|"scores rankings"| node_rank_metrics
node_runner -.->|"writes scores"| node_results
node_task_registry -.->|"registers tasks"| node_base_dataset
node_video_tasks -->|"selects task data"| node_dataset_paths
node_visual_docs -->|"selects task data"| node_dataset_paths
node_audio_tasks -->|"selects task data"| node_dataset_paths
node_audio_tasks -->|"uses instructions"| node_audio_prompts
node_agent_tasks -->|"uses dataset base"| node_base_dataset
node_mcmr_tasks -.->|"uses dataset base"| node_base_dataset
node_dataset_paths -->|"imports local map"| node_local_paths
node_audio_tasks -->|"loads datasets"| node_dataset_io
node_video_tasks -->|"loads datasets"| node_dataset_io
node_visual_docs -->|"loads datasets"| node_dataset_io
node_video_tasks -->|"uses media utilities"| node_vision
node_vision -->|"uses transforms"| node_video_transforms
node_batcher -->|"prepares model inputs"| node_processor
node_batcher -->|"processes video"| node_qwen_video
node_model -->|"uses processor"| node_processor
node_processor -->|"uses video helpers"| node_qwen_video
node_runner -->|"logs by rank"| node_rank_utils

click node_config "https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/arguments.py"
click node_runner "https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/eval.py"
click node_task_registry "https://github.com/tencent/wemm-embedding/tree/main/mmeb_v3_eval/src/data/eval_dataset"
click node_base_dataset "https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/data/eval_dataset/base_eval_dataset.py"
click node_video_tasks "https://github.com/tencent/wemm-embedding/tree/main/mmeb_v3_eval/src/data/eval_dataset"
click node_visual_docs "https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/data/eval_dataset/vidore_dataset.py"
click node_audio_tasks "https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/data/eval_dataset/audio_cls_dataset.py"
click node_agent_tasks "https://github.com/tencent/wemm-embedding/tree/main/mmeb_v3_eval/src/data/eval_dataset"
click node_mcmr_tasks "https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/data/eval_dataset/mcmr_dataset.py"
click node_dataset_paths "https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/constant/dataset_hf_path.py"
click node_local_paths "https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/constant/dataset_hflocal_path.py"
click node_model "https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/model/model.py"
click node_processor "https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/model/processor.py"
click node_batcher "https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/data/collator/eval_collator.py"
click node_vision "https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/utils/vision_utils/vision_utils.py"
click node_video_transforms "https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/utils/vision_utils/video_transforms.py"
click node_dataset_io "https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/utils/dataset_utils.py"
click node_rank_metrics "https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/utils/eval_utils/metrics.py"
click node_rank_utils "https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/utils/basic_utils.py"
click node_qwen_video "https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/model/vlm_backbone/qwen2_vl/qwen_vl_utils.py"
click node_audio_prompts "https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/data/eval_dataset/audio_instruction_utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_config,node_runner,node_results toneBlue
class node_task_registry,node_base_dataset,node_video_tasks,node_visual_docs,node_audio_tasks,node_agent_tasks,node_mcmr_tasks,node_dataset_paths,node_local_paths,node_audio_prompts toneAmber
class node_model,node_processor,node_batcher,node_qwen_video toneMint
class node_vision,node_video_transforms,node_dataset_io,node_rank_metrics,node_rank_utils toneRose
class node_user toneIndigo
```

## Components

### Benchmark orchestration

- [Run arguments](https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/arguments.py): `mmeb_v3_eval/src/arguments.py`
- [Evaluation runner](https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/eval.py): `mmeb_v3_eval/eval.py`
- **Benchmark scores**

### Task data

- [Task dataset registry](https://github.com/tencent/wemm-embedding/tree/main/mmeb_v3_eval/src/data/eval_dataset): `mmeb_v3_eval/src/data/eval_dataset`
- [Dataset base and schema](https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/data/eval_dataset/base_eval_dataset.py): `mmeb_v3_eval/src/data/eval_dataset/base_eval_dataset.py`
- [Video benchmarks](https://github.com/tencent/wemm-embedding/tree/main/mmeb_v3_eval/src/data/eval_dataset): `mmeb_v3_eval/src/data/eval_dataset`
- [Visual document tasks](https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/data/eval_dataset/vidore_dataset.py): `mmeb_v3_eval/src/data/eval_dataset/vidore_dataset.py`
- [Audio benchmarks](https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/data/eval_dataset/audio_cls_dataset.py): `mmeb_v3_eval/src/data/eval_dataset/audio_cls_dataset.py`
- [Agent and text tasks](https://github.com/tencent/wemm-embedding/tree/main/mmeb_v3_eval/src/data/eval_dataset): `mmeb_v3_eval/src/data/eval_dataset`
- [MCMR tasks](https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/data/eval_dataset/mcmr_dataset.py): `mmeb_v3_eval/src/data/eval_dataset/mcmr_dataset.py`
- [Dataset path maps](https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/constant/dataset_hf_path.py): `mmeb_v3_eval/src/constant/dataset_hf_path.py`
- [Local dataset paths](https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/constant/dataset_hflocal_path.py): `mmeb_v3_eval/src/constant/dataset_hflocal_path.py`
- [Audio instructions](https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/data/eval_dataset/audio_instruction_utils.py): `mmeb_v3_eval/src/data/eval_dataset/audio_instruction_utils.py`

### Embedding runtime

- [Embedding model](https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/model/model.py): `mmeb_v3_eval/src/model/model.py`
- [Multimodal processor](https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/model/processor.py): `mmeb_v3_eval/src/model/processor.py`
- [Evaluation collator](https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/data/collator/eval_collator.py): `mmeb_v3_eval/src/data/collator/eval_collator.py`
- [Qwen video helpers](https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/model/vlm_backbone/qwen2_vl/qwen_vl_utils.py): `mmeb_v3_eval/src/model/vlm_backbone/qwen2_vl/qwen_vl_utils.py`

### Scoring and media

- [Video and image transforms](https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/utils/vision_utils/vision_utils.py): `mmeb_v3_eval/src/utils/vision_utils/vision_utils.py`
- [Video transforms](https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/utils/vision_utils/video_transforms.py): `mmeb_v3_eval/src/utils/vision_utils/video_transforms.py`
- [Dataset loading utilities](https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/utils/dataset_utils.py): `mmeb_v3_eval/src/utils/dataset_utils.py`
- [Ranking metrics](https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/utils/eval_utils/metrics.py): `mmeb_v3_eval/src/utils/eval_utils/metrics.py`
- [Distributed logging](https://github.com/tencent/wemm-embedding/blob/main/mmeb_v3_eval/src/utils/basic_utils.py): `mmeb_v3_eval/src/utils/basic_utils.py`

### Other components

- **Evaluator**

## Connections

- Evaluator → Evaluation runner: runs evaluation (evidence: `README.md`)
- Evaluator → Run arguments: supplies settings
- Evaluation runner → Run arguments: reads settings (evidence: `mmeb_v3_eval/eval.py`)
- Evaluation runner → Embedding model: runs inference (evidence: `mmeb_v3_eval/eval.py`)
- Evaluation runner → Multimodal processor: prepares inputs (evidence: `mmeb_v3_eval/eval.py`)
- Evaluation runner → Evaluation collator: builds batches (evidence: `mmeb_v3_eval/eval.py`)
- Evaluation runner → Dataset base and schema: uses dataset base (evidence: `mmeb_v3_eval/eval.py`)
- Evaluation runner → Ranking metrics: scores rankings (evidence: `mmeb_v3_eval/eval.py`)
- Evaluation runner → Benchmark scores: writes scores
- Task dataset registry → Dataset base and schema: registers tasks (evidence: `mmeb_v3_eval/src/data/eval_dataset/complex_text_retrieve.py`)
- Video benchmarks → Dataset path maps: selects task data (evidence: `mmeb_v3_eval/src/constant/dataset_hf_path.py`)
- Visual document tasks → Dataset path maps: selects task data (evidence: `mmeb_v3_eval/src/constant/dataset_hf_path.py`)
- Audio benchmarks → Dataset path maps: selects task data (evidence: `mmeb_v3_eval/src/data/eval_dataset/audio_cls_dataset.py`)
- Audio benchmarks → Audio instructions: uses instructions (evidence: `mmeb_v3_eval/src/data/eval_dataset/audio_cls_dataset.py`)
- Agent and text tasks → Dataset base and schema: uses dataset base (evidence: `mmeb_v3_eval/src/data/eval_dataset/complex_text_retrieve.py`)
- MCMR tasks → Dataset base and schema: uses dataset base
- Dataset path maps → Local dataset paths: imports local map (evidence: `mmeb_v3_eval/src/constant/dataset_hf_path.py`)
- Audio benchmarks → Dataset loading utilities: loads datasets (evidence: `mmeb_v3_eval/src/data/eval_dataset/audio_cls_dataset.py`)
- Video benchmarks → Dataset loading utilities: loads datasets (evidence: `mmeb_v3_eval/src/data/eval_dataset/mvbench_dataset.py`)
- Visual document tasks → Dataset loading utilities: loads datasets
- Video benchmarks → Video and image transforms: uses media utilities (evidence: `mmeb_v3_eval/src/data/eval_dataset/mvbench_dataset.py`)
- Video and image transforms → Video transforms: uses transforms (evidence: `mmeb_v3_eval/src/utils/vision_utils/vision_utils.py`)
- Evaluation collator → Multimodal processor: prepares model inputs (evidence: `mmeb_v3_eval/src/data/collator/eval_collator.py`)
- Evaluation collator → Qwen video helpers: processes video (evidence: `mmeb_v3_eval/src/data/collator/eval_collator.py`)
- Embedding model → Multimodal processor: uses processor (evidence: `mmeb_v3_eval/src/model/model.py`)
- Multimodal processor → Qwen video helpers: uses video helpers (evidence: `mmeb_v3_eval/src/model/processor.py`)
- Evaluation runner → Distributed logging: logs by rank (evidence: `mmeb_v3_eval/eval.py`)
