# 3t3rn41/multimodal_rag_assistant architecture

> Architecture diagram of the GitHub repository 3t3rn41/multimodal_rag_assistant, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/3t3rn41/multimodal_rag_assistant
- Repository: https://github.com/3t3rn41/multimodal_rag_assistant

## Overview

这是一个面向文档、图片、音频和视频的多模态 RAG 助手方案。当前代码覆盖了数据解析与切片、向量化和入库，以及混合检索、重排、流式生成和引用校验；README 所列前端、后端 API、任务队列和部署仍属后续规划，故不画作已实现组件。主要流程是资料转为统一 Chunk 并索引，用户提问后检索和重排相关片段，再生成可追溯回答。媒体处理与评估能力也单独呈现。连接依据 README 和已采样源码；未采样模块仅按文档说明纳入，避免推断其内部调用。

Main entry points: [`rag_engine/pipeline.py`](https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/rag_engine/pipeline.py) (RAG 编排), [`rag_engine/retrieval.py`](https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/rag_engine/retrieval.py) (混合召回), [`ingestion/chunking.py`](https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/ingestion/chunking.py) (统一切片), [`ingestion/documents.py`](https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/ingestion/documents.py) (文档解析)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["资料处理"]
  node_document_parser["文档解析<br/>[documents.py]"]
  node_media_pipeline["媒体处理<br/>[media.py]"]
  node_chunking["统一切片<br/>[chunking.py]"]
  node_chunk_contract["Chunk 契约<br/>[models.py]"]
end

subgraph group_indexing["向量索引"]
  node_embedder["多模态向量化<br/>[api_embeddings.py]"]
  node_index_writer["批量索引<br/>[embedding.py]"]
  node_qdrant[("Qdrant 向量库")]
end

subgraph group_rag["问答生成"]
  node_pipeline["RAG 编排<br/>[pipeline.py]"]
  node_retrieval["混合召回<br/>[retrieval.py]"]
  node_qdrant_retriever["向量检索适配<br/>[qdrant_adapter.py]"]
  node_reranker["候选重排<br/>[rerank.py]"]
  node_prompt_builder["上下文提示<br/>[prompts.py]"]
  node_chat_model["流式模型适配<br/>[llm.py]"]
  node_citation_parser["引用校验<br/>[citations.py]"]
end

subgraph group_evaluation["效果评估"]
  node_evaluation_metrics["检索评估指标<br/>[metrics.py]"]
end

node_source_files(("资料文件"))
node_user(("提问者"))
node_jina_rerank{{"Jina Rerank API"}}
node_llm_service{{"兼容模型服务"}}
node_answer["回答与引用"]

node_source_files -->|"解析文档"| node_document_parser
node_source_files -->|"处理媒体"| node_media_pipeline
node_document_parser -->|"提供文本块"| node_chunking
node_media_pipeline -->|"提供媒体证据"| node_chunking
node_chunking -->|"生成 Chunk"| node_chunk_contract
node_chunk_contract -->|"提供内容"| node_embedder
node_embedder -->|"生成向量"| node_index_writer
node_index_writer -->|"写入向量"| node_qdrant
node_user -->|"提交问题"| node_pipeline
node_pipeline -->|"发起召回"| node_retrieval
node_retrieval -->|"请求向量检索"| node_qdrant_retriever
node_qdrant_retriever -->|"查询向量"| node_qdrant
node_retrieval -->|"融合候选并重排"| node_reranker
node_reranker -.->|"请求评分"| node_jina_rerank
node_pipeline -->|"构建上下文"| node_prompt_builder
node_pipeline -->|"请求生成"| node_chat_model
node_chat_model -.->|"流式调用"| node_llm_service
node_pipeline -->|"校验引用"| node_citation_parser
node_pipeline -->|"输出回答"| node_answer
node_retrieval -.->|"提供检索结果"| node_evaluation_metrics
node_citation_parser -.->|"提供引用结果"| node_evaluation_metrics

click node_document_parser "https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/ingestion/documents.py"
click node_media_pipeline "https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/ingestion/media.py"
click node_chunking "https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/ingestion/chunking.py"
click node_chunk_contract "https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/rag_engine/models.py"
click node_embedder "https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/ingestion/api_embeddings.py"
click node_index_writer "https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/ingestion/embedding.py"
click node_pipeline "https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/rag_engine/pipeline.py"
click node_retrieval "https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/rag_engine/retrieval.py"
click node_qdrant_retriever "https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/rag_engine/qdrant_adapter.py"
click node_reranker "https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/rag_engine/rerank.py"
click node_prompt_builder "https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/rag_engine/prompts.py"
click node_chat_model "https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/rag_engine/llm.py"
click node_citation_parser "https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/rag_engine/citations.py"
click node_evaluation_metrics "https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/evaluation/metrics.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_document_parser,node_media_pipeline,node_chunking,node_chunk_contract toneBlue
class node_embedder,node_index_writer,node_qdrant toneAmber
class node_pipeline,node_retrieval,node_qdrant_retriever,node_reranker,node_prompt_builder,node_chat_model,node_citation_parser,node_jina_rerank toneMint
class node_evaluation_metrics toneRose
class node_source_files,node_user,node_llm_service,node_answer toneIndigo
```

## Components

### 资料处理

- [文档解析](https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/ingestion/documents.py): `ingestion/documents.py`
- [媒体处理](https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/ingestion/media.py): `ingestion/media.py`
- [统一切片](https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/ingestion/chunking.py): `ingestion/chunking.py`
- [Chunk 契约](https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/rag_engine/models.py): `rag_engine/models.py`

### 向量索引

- [多模态向量化](https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/ingestion/api_embeddings.py): `ingestion/api_embeddings.py`
- [批量索引](https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/ingestion/embedding.py): `ingestion/embedding.py`
- **Qdrant 向量库**

### 问答生成

- [RAG 编排](https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/rag_engine/pipeline.py): `rag_engine/pipeline.py`
- [混合召回](https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/rag_engine/retrieval.py): `rag_engine/retrieval.py`
- [向量检索适配](https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/rag_engine/qdrant_adapter.py): `rag_engine/qdrant_adapter.py`
- [候选重排](https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/rag_engine/rerank.py): `rag_engine/rerank.py`
- [上下文提示](https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/rag_engine/prompts.py): `rag_engine/prompts.py`
- [流式模型适配](https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/rag_engine/llm.py): `rag_engine/llm.py`
- [引用校验](https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/rag_engine/citations.py): `rag_engine/citations.py`

### 效果评估

- [检索评估指标](https://github.com/3t3rn41/multimodal_rag_assistant/blob/main/evaluation/metrics.py): `evaluation/metrics.py`

### Other components

- **资料文件**
- **提问者**
- **Jina Rerank API**
- **兼容模型服务**
- **回答与引用**

## Connections

- 资料文件 → 文档解析: 解析文档
- 资料文件 → 媒体处理: 处理媒体
- 文档解析 → 统一切片: 提供文本块
- 媒体处理 → 统一切片: 提供媒体证据
- 统一切片 → Chunk 契约: 生成 Chunk
- Chunk 契约 → 多模态向量化: 提供内容
- 多模态向量化 → 批量索引: 生成向量
- 批量索引 → Qdrant 向量库: 写入向量
- 提问者 → RAG 编排: 提交问题
- RAG 编排 → 混合召回: 发起召回
- 混合召回 → 向量检索适配: 请求向量检索
- 向量检索适配 → Qdrant 向量库: 查询向量
- 混合召回 → 候选重排: 融合候选并重排
- 候选重排 → Jina Rerank API: 请求评分
- RAG 编排 → 上下文提示: 构建上下文
- RAG 编排 → 流式模型适配: 请求生成
- 流式模型适配 → 兼容模型服务: 流式调用
- RAG 编排 → 引用校验: 校验引用
- RAG 编排 → 回答与引用: 输出回答
- 混合召回 → 检索评估指标: 提供检索结果
- 引用校验 → 检索评估指标: 提供引用结果