How bytedance-seed/m3-agent works
M3-Agent has two principal workflows: it turns video and audio into multimodal episodic/semantic memory, then answers instructions through iterative retrieval and reasoning over that memory. The repository also includes M3-Bench annotations for evaluating long-video question answering. Source-index references support the orchestration and module links below; the README establishes the high-level memory/control relationship. Several implementation details are unsampled, so those links are documented rather than inferred from module names.
Main entry points: memorization_memory_graphs.py Memory graph generation · memorization_intermediate_outputs.py Intermediate generation · memory_processing_qwen.py Memory generation · control.py Control agent
Components 19
Media perception
- Intermediate generation
m3_agent/memorization_intermediate_outputs.py - Video processing
mmagent/utils/video_processing.py - Face processing
mmagent/face_processing.py - Face extraction
mmagent/src/face_extraction.py - Face clustering
mmagent/src/face_clustering.py - Voice processing
mmagent/voice_processing.py
Memory construction
- Memory graph generation
m3_agent/memorization_memory_graphs.py - Memory generation
mmagent/memory_processing_qwen.py - Memory processing
mmagent/memory_processing.py - Task prompts
mmagent/prompts.py - Multimodal memory graph
mmagent/videograph.py
Control and retrieval
- Control agent
m3_agent/control.py - Memory retrieval
mmagent/retrieve.py - Chat and embeddings
mmagent/utils/chat_api.py
Benchmark
- M3-Bench annotations
data/annotations/robot.json
Other components
- Video input
- Instruction
- Model API
- Answer
Connections 24
- Video input → Intermediate generation: submits video
README.md - Intermediate generation → Video processing: processes media
m3_agent/memorization_intermediate_outputs.py - Intermediate generation → Face processing: extracts faces
m3_agent/memorization_intermediate_outputs.py - Intermediate generation → Voice processing: processes voices
m3_agent/memorization_intermediate_outputs.py - Face processing → Face extraction: detects faces
mmagent/face_processing.py - Face processing → Face clustering: clusters faces
mmagent/face_processing.py - Intermediate generation → Memory graph generation: supplies outputs
README.md - Video input → Memory graph generation: generates memory
README.md - Memory graph generation → Video processing: processes clips
m3_agent/memorization_memory_graphs.py - Memory graph generation → Face processing: uses face data
m3_agent/memorization_memory_graphs.py - Memory graph generation → Voice processing: uses voice data
m3_agent/memorization_memory_graphs.py - Memory graph generation → Memory generation: generates memories
m3_agent/memorization_memory_graphs.py - Memory generation → Memory processing: reuses processing
mmagent/memory_processing_qwen.py - Memory generation → Task prompts: uses prompts
mmagent/memory_processing_qwen.py - Memory generation → Multimodal memory graph: updates graph
mmagent/memory_processing_qwen.py - Memory processing → Multimodal memory graph: builds graph memories
mmagent/memory_processing.py - Instruction → Control agent: asks question
README.md - Control agent → Memory retrieval: requests retrieval
m3_agent/control.py - Control agent → Chat and embeddings: reasons with model
m3_agent/control.py - Memory retrieval → Chat and embeddings: embeds queries
mmagent/retrieve.py - Chat and embeddings → Model API: calls API
mmagent/utils/chat_api.py - Multimodal memory graph → Memory retrieval: provides memory
README.md - Control agent → Answer: returns answer
m3_agent/control.py - M3-Bench annotations → Instruction: evaluates questions
README.md