# microsoft/pike-rag architecture

> Architecture diagram of the GitHub repository microsoft/pike-rag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/pike-rag
- Repository: https://github.com/microsoft/pike-rag

## Overview

##### Project Type & Purpose
- PIKE-RAG is a specialized retrieval-augmented generation (RAG) framework. Its purpose is to enhance LLM outputs by integrating domain-specific knowledge extraction, document parsing, and multi-step reasoning.
- The system is organized into several modules that handle document loading, knowledge extraction, knowledge retrieval, and prompt-based reasoning. It is used to power pipelines that are configurable for various tasks (e.g., multi-hop question answering).

##### Analyzing the File Structure
- Top-level directories like “pikerag”, “data_process”, “examples”, “docs”, and “docker_files” indicate a modular design.
- The “pikerag” directory contains key modules:
- document_loaders and document_transformers: handling data ingestion and formatting.
- knowledge_retrievers: which includes several retrieval strategies (BM25, Chroma, etc.) and query parsing utilities.
- llm_client: interfaces with different LLM providers (Azure, HuggingFace).
- prompts: containing all the logic for different prompting techniques (chunking, decomposition, tagging).
- workflows: orchestrating pipeline steps for QA, evaluation, and other processing.
- The “data_process” directory provides scripts for parsing raw data and preparing benchmark datasets.
- “examples” contains sample configurations and pipelines (e.g., for HotpotQA, MuSiQue) to demonstrate how the system is to be used.
- The presence of configuration files (YAML) and docker files indicates support for both dynamic pipeline configuration and containerized deployment.

##### Key Architectural Elements to Capture
a. Main Components:
- Data Input & Preprocessing:
▫ Data Ingestion and Processing modules (e.g., scripts in data_process) which pre-format datasets.
▫ Document Loading and Parsing (found in pikerag/document_loaders).
- Knowledge Extraction and Transformation:
▫ Document Transformers (e.g., filters, splitters, taggers) under pikerag/document_transformers.
▫ Various specialized prompt modules (chunking, decomposition, tagging) that help in preparing the knowledge base.
- Knowledge Retrieval:
▫ Multiple knowledge retrieval components (BM25, Chroma, hybrid methods) under pikerag/knowledge_retrievers.
- LLM Integration:
▫ LLM Client modules (in pikerag/llm_client) that abstract calls to external LLM services.
- Workflow Orchestration:
▫ Workflows (found in pikerag/workflows) that coordinate the overall end-to-end pipelines (QA, evaluation, tagging).
- Supporting Components:
▫ Utility functions, configuration loader, logging etc., spread across utilities and scripts.
▫ Deployment and containerization support reflected in docker_files.
b. Relationships & Interactions:
- Data flows from ingestion (document loaders) → transformation (document transformers) → knowledge storage.
- Retrieval components then query processed and stored knowledge to prepare prompts.
- The LLM client integrates with these components via the workflows to generate responses.
- Configuration files (YAML) guide the behavior of each module; this allows for plugging in alternate strategies (e.g., different retrieval backends or prompt templates).
c. Architectural Patterns:
- Modular design: encapsulating different responsibilities (data handling, transformation, retrieval, reasoning) in separate directories.
- Pipeline orchestration: series of discrete steps chain together to form a full end-to-end response generation framework.
- Extensibility: use of configuration files and multiple implementations (plugins) for document processing and retrieval.
d. Relevant Technologies & Tools:
- Python as the primary language.
- YAML for configuration.
- Docker for deployment.
- Third-party APIs/integrations for LLMs (Azure OpenAI, HuggingFace).
- Likely use of libraries for natural language processing and retrieval (e.g., BM25, Chroma).

- Next, a “Processing Layer” that hosts knowledge extraction components (document transformers, prompt generation, filtering, splitting).
- Follow it with a “Retrieval Layer” that retrieves relevant knowledge using various underlying strategies (BM25, Chroma).
- Then, an “LLM Integration Layer” where the LLM client receives the prepared prompts to generate answers.
- Finally, a “Workflow/Orchestration” component that ties everything together and coordinates the entire pipeline.
- Surrounding the core with auxiliary components (deployment configurations, example pipelines, and diagnostics/logging).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input Layer
    subgraph "Input Layer"
        DL["Document Loaders"]:::ingestion
        DPS["Data Process Scripts"]:::ingestion
    end

    %% Processing Layer
    subgraph "Processing Layer"
        DT["Document Transformers"]:::processing
        PM["Prompt Modules"]:::processing
    end

    %% Retrieval Layer
    subgraph "Retrieval Layer"
        KR["Knowledge Retrievers"]:::retrieval
    end

    %% LLM Integration Layer
    subgraph "LLM Integration"
        LLM["LLM Client"]:::llm
    end

    %% Workflow Orchestration Layer
    subgraph "Workflow Orchestration"
        WF["Workflows"]:::workflow
    end

    %% Deployment
    subgraph "Deployment"
        DC["Containerization"]:::deployment
    end

    %% Data Flow Connections
    DL -->|"to"| DT
    DPS -->|"to"| DT
    DT -->|"to"| PM
    PM -->|"to"| KR
    KR -->|"to"| LLM
    LLM -->|"to"| WF
    WF -->|"deploys"| DC

    %% Click Events
    click DL "https://github.com/microsoft/pike-rag/tree/main/pikerag/document_loaders"
    click DPS "https://github.com/microsoft/pike-rag/tree/main/data_process"
    click DT "https://github.com/microsoft/pike-rag/tree/main/pikerag/document_transformers"
    click PM "https://github.com/microsoft/pike-rag/tree/main/pikerag/prompts"
    click KR "https://github.com/microsoft/pike-rag/tree/main/pikerag/knowledge_retrievers"
    click LLM "https://github.com/microsoft/pike-rag/tree/main/pikerag/llm_client"
    click WF "https://github.com/microsoft/pike-rag/tree/main/pikerag/workflows"
    click DC "https://github.com/microsoft/pike-rag/tree/main/docker_files"

    %% Class Definitions for Colors
    classDef ingestion fill:#FDEBD0,stroke:#F5B041,stroke-width:2px;
    classDef processing fill:#D5F5E3,stroke:#82E0AA,stroke-width:2px;
    classDef retrieval fill:#D6EAF8,stroke:#3498DB,stroke-width:2px;
    classDef llm fill:#FAD7A0,stroke:#F39C12,stroke-width:2px;
    classDef workflow fill:#E8DAEF,stroke:#8E44AD,stroke-width:2px;
    classDef deployment fill:#FCE4EC,stroke:#EC407A,stroke-width:2px;
```
