# bytedance-seed/m3-agent architecture

> Architecture diagram of the GitHub repository bytedance-seed/m3-agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bytedance-seed/m3-agent
- Repository: https://github.com/bytedance-seed/m3-agent

## Overview

M3-Agent has two principal workflows: it turns video and audio into multimodal episodic/semantic memory, then answers instructions through iterative retrieval and reasoning over that memory. The repository also includes M3-Bench annotations for evaluating long-video question answering. Source-index references support the orchestration and module links below; the README establishes the high-level memory/control relationship. Several implementation details are unsampled, so those links are documented rather than inferred from module names.

Main entry points: [`m3_agent/memorization_memory_graphs.py`](https://github.com/bytedance-seed/m3-agent/blob/master/m3_agent/memorization_memory_graphs.py) (Memory graph generation), [`m3_agent/memorization_intermediate_outputs.py`](https://github.com/bytedance-seed/m3-agent/blob/master/m3_agent/memorization_intermediate_outputs.py) (Intermediate generation), [`mmagent/memory_processing_qwen.py`](https://github.com/bytedance-seed/m3-agent/blob/master/mmagent/memory_processing_qwen.py) (Memory generation), [`m3_agent/control.py`](https://github.com/bytedance-seed/m3-agent/blob/master/m3_agent/control.py) (Control agent)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_perception["Media perception"]
  node_intermediate_job["Intermediate generation"]
  node_video_processing["Video processing"]
  node_face_processing["Face processing<br/>[face_processing.py]"]
  node_face_extract["Face extraction<br/>[face_extraction.py]"]
  node_face_cluster["Face clustering<br/>[face_clustering.py]"]
  node_voice_processing["Voice processing"]
end

subgraph group_memory["Memory construction"]
  node_memory_job["Memory graph generation"]
  node_memory_model["Memory generation"]
  node_memory_processing["Memory processing"]
  node_prompts["Task prompts<br/>[prompts.py]"]
  node_video_graph[("Multimodal memory graph<br/>[videograph.py]")]
end

subgraph group_control["Control and retrieval"]
  node_control_agent["Control agent<br/>[control.py]"]
  node_retrieval["Memory retrieval<br/>[retrieve.py]"]
  node_chat_api["Chat and embeddings<br/>[chat_api.py]"]
end

subgraph group_benchmark["Benchmark"]
  node_benchmark_data["M3-Bench annotations<br/>[robot.json]"]
end

node_video_user(("Video input"))
node_instruction_user(("Instruction"))
node_chat_service{{"Model API"}}
node_answer(("Answer"))

node_video_user -->|"submits video"| node_intermediate_job
node_intermediate_job -->|"processes media"| node_video_processing
node_intermediate_job -->|"extracts faces"| node_face_processing
node_intermediate_job -->|"processes voices"| node_voice_processing
node_face_processing -->|"detects faces"| node_face_extract
node_face_processing -->|"clusters faces"| node_face_cluster
node_intermediate_job -->|"supplies outputs"| node_memory_job
node_video_user -->|"generates memory"| node_memory_job
node_memory_job -->|"processes clips"| node_video_processing
node_memory_job -->|"uses face data"| node_face_processing
node_memory_job -->|"uses voice data"| node_voice_processing
node_memory_job -->|"generates memories"| node_memory_model
node_memory_model -->|"reuses processing"| node_memory_processing
node_memory_model -->|"uses prompts"| node_prompts
node_memory_model -->|"updates graph"| node_video_graph
node_memory_processing -->|"builds graph memories"| node_video_graph
node_instruction_user -->|"asks question"| node_control_agent
node_control_agent -->|"requests retrieval"| node_retrieval
node_control_agent -->|"reasons with model"| node_chat_api
node_retrieval -->|"embeds queries"| node_chat_api
node_chat_api -->|"calls API"| node_chat_service
node_video_graph -->|"provides memory"| node_retrieval
node_control_agent -->|"returns answer"| node_answer
node_benchmark_data -.->|"evaluates questions"| node_instruction_user

click node_intermediate_job "https://github.com/bytedance-seed/m3-agent/blob/master/m3_agent/memorization_intermediate_outputs.py"
click node_video_processing "https://github.com/bytedance-seed/m3-agent/blob/master/mmagent/utils/video_processing.py"
click node_face_processing "https://github.com/bytedance-seed/m3-agent/blob/master/mmagent/face_processing.py"
click node_face_extract "https://github.com/bytedance-seed/m3-agent/blob/master/mmagent/src/face_extraction.py"
click node_face_cluster "https://github.com/bytedance-seed/m3-agent/blob/master/mmagent/src/face_clustering.py"
click node_voice_processing "https://github.com/bytedance-seed/m3-agent/blob/master/mmagent/voice_processing.py"
click node_memory_job "https://github.com/bytedance-seed/m3-agent/blob/master/m3_agent/memorization_memory_graphs.py"
click node_memory_model "https://github.com/bytedance-seed/m3-agent/blob/master/mmagent/memory_processing_qwen.py"
click node_memory_processing "https://github.com/bytedance-seed/m3-agent/blob/master/mmagent/memory_processing.py"
click node_prompts "https://github.com/bytedance-seed/m3-agent/blob/master/mmagent/prompts.py"
click node_video_graph "https://github.com/bytedance-seed/m3-agent/blob/master/mmagent/videograph.py"
click node_control_agent "https://github.com/bytedance-seed/m3-agent/blob/master/m3_agent/control.py"
click node_retrieval "https://github.com/bytedance-seed/m3-agent/blob/master/mmagent/retrieve.py"
click node_chat_api "https://github.com/bytedance-seed/m3-agent/blob/master/mmagent/utils/chat_api.py"
click node_benchmark_data "https://github.com/bytedance-seed/m3-agent/blob/master/data/annotations/robot.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_intermediate_job,node_video_processing,node_face_processing,node_face_extract,node_face_cluster,node_voice_processing toneBlue
class node_memory_job,node_memory_model,node_memory_processing,node_prompts,node_video_graph toneAmber
class node_control_agent,node_retrieval,node_chat_api,node_chat_service toneMint
class node_benchmark_data toneRose
class node_video_user,node_instruction_user,node_answer toneIndigo
```

## Components

### Media perception

- [Intermediate generation](https://github.com/bytedance-seed/m3-agent/blob/master/m3_agent/memorization_intermediate_outputs.py): `m3_agent/memorization_intermediate_outputs.py`
- [Video processing](https://github.com/bytedance-seed/m3-agent/blob/master/mmagent/utils/video_processing.py): `mmagent/utils/video_processing.py`
- [Face processing](https://github.com/bytedance-seed/m3-agent/blob/master/mmagent/face_processing.py): `mmagent/face_processing.py`
- [Face extraction](https://github.com/bytedance-seed/m3-agent/blob/master/mmagent/src/face_extraction.py): `mmagent/src/face_extraction.py`
- [Face clustering](https://github.com/bytedance-seed/m3-agent/blob/master/mmagent/src/face_clustering.py): `mmagent/src/face_clustering.py`
- [Voice processing](https://github.com/bytedance-seed/m3-agent/blob/master/mmagent/voice_processing.py): `mmagent/voice_processing.py`

### Memory construction

- [Memory graph generation](https://github.com/bytedance-seed/m3-agent/blob/master/m3_agent/memorization_memory_graphs.py): `m3_agent/memorization_memory_graphs.py`
- [Memory generation](https://github.com/bytedance-seed/m3-agent/blob/master/mmagent/memory_processing_qwen.py): `mmagent/memory_processing_qwen.py`
- [Memory processing](https://github.com/bytedance-seed/m3-agent/blob/master/mmagent/memory_processing.py): `mmagent/memory_processing.py`
- [Task prompts](https://github.com/bytedance-seed/m3-agent/blob/master/mmagent/prompts.py): `mmagent/prompts.py`
- [Multimodal memory graph](https://github.com/bytedance-seed/m3-agent/blob/master/mmagent/videograph.py): `mmagent/videograph.py`

### Control and retrieval

- [Control agent](https://github.com/bytedance-seed/m3-agent/blob/master/m3_agent/control.py): `m3_agent/control.py`
- [Memory retrieval](https://github.com/bytedance-seed/m3-agent/blob/master/mmagent/retrieve.py): `mmagent/retrieve.py`
- [Chat and embeddings](https://github.com/bytedance-seed/m3-agent/blob/master/mmagent/utils/chat_api.py): `mmagent/utils/chat_api.py`

### Benchmark

- [M3-Bench annotations](https://github.com/bytedance-seed/m3-agent/blob/master/data/annotations/robot.json): `data/annotations/robot.json`

### Other components

- **Video input**
- **Instruction**
- **Model API**
- **Answer**

## Connections

- Video input → Intermediate generation: submits video (evidence: `README.md`)
- Intermediate generation → Video processing: processes media (evidence: `m3_agent/memorization_intermediate_outputs.py`)
- Intermediate generation → Face processing: extracts faces (evidence: `m3_agent/memorization_intermediate_outputs.py`)
- Intermediate generation → Voice processing: processes voices (evidence: `m3_agent/memorization_intermediate_outputs.py`)
- Face processing → Face extraction: detects faces (evidence: `mmagent/face_processing.py`)
- Face processing → Face clustering: clusters faces (evidence: `mmagent/face_processing.py`)
- Intermediate generation → Memory graph generation: supplies outputs (evidence: `README.md`)
- Video input → Memory graph generation: generates memory (evidence: `README.md`)
- Memory graph generation → Video processing: processes clips (evidence: `m3_agent/memorization_memory_graphs.py`)
- Memory graph generation → Face processing: uses face data (evidence: `m3_agent/memorization_memory_graphs.py`)
- Memory graph generation → Voice processing: uses voice data (evidence: `m3_agent/memorization_memory_graphs.py`)
- Memory graph generation → Memory generation: generates memories (evidence: `m3_agent/memorization_memory_graphs.py`)
- Memory generation → Memory processing: reuses processing (evidence: `mmagent/memory_processing_qwen.py`)
- Memory generation → Task prompts: uses prompts (evidence: `mmagent/memory_processing_qwen.py`)
- Memory generation → Multimodal memory graph: updates graph (evidence: `mmagent/memory_processing_qwen.py`)
- Memory processing → Multimodal memory graph: builds graph memories (evidence: `mmagent/memory_processing.py`)
- Instruction → Control agent: asks question (evidence: `README.md`)
- Control agent → Memory retrieval: requests retrieval (evidence: `m3_agent/control.py`)
- Control agent → Chat and embeddings: reasons with model (evidence: `m3_agent/control.py`)
- Memory retrieval → Chat and embeddings: embeds queries (evidence: `mmagent/retrieve.py`)
- Chat and embeddings → Model API: calls API (evidence: `mmagent/utils/chat_api.py`)
- Multimodal memory graph → Memory retrieval: provides memory (evidence: `README.md`)
- Control agent → Answer: returns answer (evidence: `m3_agent/control.py`)
- M3-Bench annotations → Instruction: evaluates questions (evidence: `README.md`)
