How hkuds/minirag works
The MiniRAG project is a Retrieval-Augmented Generation framework designed for on-device RAG systems. It is focused on enabling small language models to perform efficient retrieval and generation over a heterogeneous graph index. The project leverages lightweight graph-based knowledge retrieval and supports multiple graph database backends, various language model interfaces, and deployment options such as API and Docker-based setups.
Project Type and Purpose
• MiniRAG is a specialized full-stack library focused on retrieval-augmented generation. Its core purpose is to enable small language models to benefit from graph-based indexing and retrieval (heterogeneous graph indexing) to boost performance in RAG tasks.
• The project is experimental/research-oriented with production-grade deployment options (API, Docker) and includes benchmark datasets (like LiHua-World).
Analyzing the File Structure
• Top-Level Files:
• .gitignore & .pre-commit-config.yaml: Set up version control exclusions and pre-commit hook configurations.
• Dockerfile & docker-compose.yml: Provide containerization and multi-container orchestration instructions for deployment.
• LICENSE, MANIFEST.in, pyproject.toml, requirements.txt, setup.py: Files for legal, packaging, dependency management, and installation purposes.
• README.md, README_CN.md, README_JA.md: Provide multilingual documentation on usage, architecture, and system details.
• main.py: Likely an application entry point for utilizing MiniRAG.
• Directories:
• assets, graph-visuals: These directories store static assets and visualization scripts respectively.
• dataset: Contains the LiHua-World dataset including data files, query sets (CSV/JSON), and notebooks (evaluation.ipynb) that explain how data is organized and used in training or testing.
• reproduce: Holds reproduction scripts (e.g., Step_0_index.py, Step_1_QA.py) used to perform indexing and question-answering experiments.
• minirag: The core source package that comprises:
• api: Contains the API server code (minirag_server.py) along with environment configurations, static files (HTML/JavaScript), and setup documentation (README.md). This supports API and Docker deployment.
• base.py: Defines the base classes or core logic shared across the project.
• exceptions.py: Contains custom exception definitions.
• kg: A subpackage listing multiple implementations (e.g., neo4j_impl.py, oracle_impl.py, redis_impl.py, etc.) that serve as interfaces to different graph databases for heterogeneous graph indexing.
• llm & llm.py: These files provide support for interfacing with different language model services (azure_openai, cuffing with platforms like OpenAI, Hugging Face, etc.) and manage the generation component of the RAG system.
• minirag.py: Likely orchestrates or integrates various modules (retrieval, generation, and data processing) for the complete workflow.
• operate.py, prompt.py, utils.py: Supplementary modules for operational tasks, prompt management, and utility/helper functions.
Detailed File-Level Explanation
• .gitignore, .pre-commit-config.yaml: Define ignore patterns and enforce code quality through pre-commit hooks.
• Dockerfile, docker-compose.yml: Provide instructions and configurations for containerizing and orchestrating the application.
• LICENSE, MANIFEST.in, pyproject.toml, requirements.txt, setup.py: Manage licensing, package inclusion, dependency management, and project installation.
• README.md (and its translations in README_CN.md, README_JA.md): Document the project’s purpose, usage, and technical details.
• assets, graph-visuals: Store visual assets and scripts for visualizing the underlying graph structures – useful for understanding graph-based retrieval.
• dataset/LiHua-World: Contains raw data (in LiHuaWorld.zip), evaluation notebooks, and QA files (CSV and JSON) for testing the retrieval performance.
• main.py: Likely orchestrates a high-level execution path to initialize indexing and query-answering.
• minirag/
• __init__.py: Marks the directory as a Python module.
• base.py: Provides shared foundational logic.
• exceptions.py: Contains custom exception classes to handle errors.
• kg/ (e.g., neo4j_impl.py, oracle_impl.py, etc.): Each file implements an interface to a specific graph database for heterogeneous indexing.
• llm/ (e.g., azure_openai.py, bedrock.py, etc.): Each script provides integration with different language model services, forming the generation part of the RAG system.
• minirag.py: Integrates the main components (retrieval and generation) to implement the MiniRAG workflow.
• operate.py: May cover operational tasks such as orchestration or module integration.
• prompt.py: Likely holds prompt templates and example queries for use with the LLMs.
• utils.py: Contains helper functions used throughout the project.
• reproduce/ (Step_0_index.py, Step_1_QA.py): Scripts to reproduce the indexing and QA experiments, useful for benchmarking and testing system performance.