How hkuds/rag-anything works
Purpose and Entry Points
RAG-Anything is a Python 3.10 multimodal document RAG library built on LightRAG. It ingests documents or pre-parsed content, derives text and modality-specific semantic representations, indexes them through LightRAG, and answers text or VLM-enhanced queries. Primary public API: RAGAnything and configuration exports in raganything/__init__.py; orchestration resides in raganything/raganything.py.
Core Orchestration and LightRAG Boundary
RAGAnything owns pipeline coordination while delegating persistence, embeddings, graph construction, vector retrieval, and baseline query modes to an injected or internally constructed LightRAG instance. Callers supply LLM, embedding, and optional vision-model functions; this keeps provider choice outside the library. Configuration and shared domain abstractions are in raganything/config.py, raganything/base.py, and raganything/raganything.py.
Ingestion and Parser Boundary
Document ingestion accepts PDFs, Office files, images, text, audio/video, folders, or direct content_list insertion. The parser layer normalizes parser-specific output into the flat content-list contract, preserving page/order, captions, footnotes, paths, and structure. MinerU, Docling, and PaddleOCR are selectable external parser backends; MinerU output includes v2-to-legacy normalization. Key paths: raganything/parser.py, raganything/mineru_content.py, raganything/batch_parser.py.
Content Routing and Processing
The processor splits normalized content into text chunks and modality work, retaining source-document identity and hierarchy. Text flows to LightRAG insertion; image, table, equation, generic, audio, and video items route to dedicated processors that generate descriptions and entities/chunks for the same knowledge base. Main paths: raganything/processor.py, raganything/modalprocessors.py, raganything/modalprocessors_audio.py, raganything/modalprocessors_video.py.
Multimodal Enrichment and Graph Indexing
Modality processors use caller-provided caption/VLM functions to interpret non-text assets, then create indexed chunks and graph entities/relationships through LightRAG. This is the critical integration boundary: RAG-Anything enriches content and preserves cross-modal/document relationships, while LightRAG owns the graph/vector backends. Prompts and language-specific prompt management are centralized in raganything/prompt.py, raganything/prompt_manager.py, and raganything/prompts_zh.py.
Query Paths
aquery/query forward text retrieval modes—hybrid, local, global, naive—to LightRAG. When enabled, query logic identifies retrieved image context, loads/encodes assets, and sends text plus images to the vision model. aquery_with_multimodal accepts caller-provided tables, equations, or other modal context alongside retrieval. Query assembly is in raganything/query.py; public coordination remains in raganything/raganything.py.
Batch, Reliability, and Runtime Dependencies
Batch operations parallelize folder ingestion with dry-run/incremental behavior: raganything/batch.py and raganything/batch_parser.py. Resilience and callbacks provide operational hooks around asynchronous work: raganything/resilience.py, raganything/callbacks.py. Runtime infrastructure is Python asyncio plus LightRAG storage backends (file-based by default, externally configurable alternatives such as Neo4j). External services/tools include LLM/embedding/VLM APIs (OpenAI-compatible, Ollama, vLLM, LM Studio, MiniMax examples), MinerU/Docling/PaddleOCR, LibreOffice for Office conversion, faster-whisper for audio, SceneDetect and ffmpeg for video, and optional CDN/object-storage URL mapping via raganything/asset_urls.py.
Main entry points: raganything.py RAGAnything · parser.py Parser facade · processor.py Content processor · modalprocessors.py Image/table/equation processors
Components 18
Core API
- Public exports · Exports RAGAnything and configuration.
raganything/__init__.py - RAGAnything · Coordinates ingestion, indexing, and query flows.
raganything/raganything.py - Config & domain abstractions
raganything/config.py - Resilience & callbacks · Provides asynchronous reliability handling and operational callbacks.
raganything/resilience.py
Ingestion
- Parser facade · Converts source documents into the flat content-list contract.
raganything/parser.py - MinerU normalization · Normalizes MinerU v2 output to the legacy content shape.
raganything/mineru_content.py - Batch ingestion · Supports dry-run and incremental folder processing.
raganything/batch.py - Batch parser
raganything/batch_parser.py
Multimodal Enrichment
- Content processor · Splits text chunks and routes normalized modal items.
raganything/processor.py - Image/table/equation processors · Creates descriptions, chunks, and semantic entities for visual content.
raganything/modalprocessors.py - Audio processor
raganything/modalprocessors_audio.py - Video processor
raganything/modalprocessors_video.py - Prompt management · Centralizes prompts and language-specific prompt selection.
raganything/prompt_manager.py
Query
- Query assembly · Builds retrieval queries and attaches retrieved image context for VLM calls.
raganything/query.py
External Boundaries
- LightRAG · Owns persistence, embeddings, graph construction, vector retrieval, and baseline query modes.
- LLM, embedding & vision functions · Provider selection remains outside the library.
- Parser backends · MinerU, Docling, and PaddleOCR parse source documents.
- Media & conversion tools · LibreOffice, faster-whisper, SceneDetect, and ffmpeg support source conversion.
Connections 26
- Public exports → RAGAnything: exports
- Config & domain abstractions → RAGAnything: configures
- RAGAnything → Parser facade: ingests documents
- RAGAnything → Content processor: inserts content lists
- Batch ingestion → Batch parser: parallelizes parsing
- Batch ingestion → RAGAnything: submits ingestion
- Batch parser → Parser facade: uses
- Parser facade → MinerU normalization: normalizes MinerU
- Parser facade → Parser backends: delegates parsing
- Parser facade → Media & conversion tools: converts media sources
- Parser facade → Content processor: normalized content list
- Content processor → LightRAG: indexes text chunks
- Content processor → Image/table/equation processors: routes visual items
- Content processor → Audio processor: routes audio
- Content processor → Video processor: routes video
- Image/table/equation processors → LightRAG: indexes enriched chunks and graph links
- Audio processor → LightRAG: indexes enrichment
- Video processor → LightRAG: indexes enrichment
- Image/table/equation processors → LLM, embedding & vision functions: uses caption/VLM
- Audio processor → LLM, embedding & vision functions: uses models
- Video processor → LLM, embedding & vision functions: uses models
- Prompt management → Image/table/equation processors: supplies prompts
- RAGAnything → Query assembly: aquery/query
- Query assembly → LightRAG: retrieves hybrid/local/global/naive
- Query assembly → LLM, embedding & vision functions: sends text and image context
- Resilience & callbacks → RAGAnything: wraps async operations