# microsoft/rag-experiment-accelerator architecture

> Architecture diagram of the GitHub repository microsoft/rag-experiment-accelerator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/rag-experiment-accelerator
- Repository: https://github.com/microsoft/rag-experiment-accelerator

## Overview

##### Project Type & Purpose
- This is an open-source experimentation accelerator for Retrieval-Augmented Generation (RAG) built in Python.
- It provides both CLI scripts (01_index.py → 04_evaluation.py) and orchestrated pipelines (Azure ML, PromptFlow) to ingest documents, build/search indexes, generate Q&A, rerank results, and compute metrics.
- It integrates with Azure AI Search, Azure Document Intelligence, Azure OpenAI, Azure Machine Learning, MLFlow, and local data.

##### File Structure Analysis
- Top-level scripts: 01_index.py, 02_qa_generation.py, 03_querying.py, 04_evaluation.py (CLI entrypoints).
- azureml/: Azure ML SDK pipeline definitions and helper scripts.
- promptflow/: PromptFlow DAGs and custom environments to orchestrate the same four steps in a serverless fashion.
- infra/: Bicep/ARM templates to provision Azure resources (Search, Key Vault, ML, Storage, Cognitive Services).
- rag_experiment_accelerator/: Core Python library organized into submodules:
- config (load/validate experiment settings)
- doc_loader (multi-format document ingestion & chunking)
- embedding (create vector embeddings via Azure OAI or sentence-transformers)
- io (local file read/write abstractions & checkpointing)
- llm (prompt construction and response generation)
- reranking (LLM or cross-encoder based reranking)
- evaluation (end-to-end & component metrics, plotting, MLFlow integration)
- sampling (content-driven clustering & selection)
- checkpoint (skip reprocessing of completed steps)
- data/: Sample documents; artifacts/ for sampling outputs.
- config.sample.json / .env.template: config and environment variable blueprints.

##### Core Components & Interactions
- Development Environment (local or DevContainer) → orchestrates either CLI scripts, Azure ML pipeline, or PromptFlow flows.
- Document Ingestion (doc_loader) → produces chunks.
- Indexing (rag_experiment_accelerator.run.index & azureml/index.py & promptflow/index) → uses embedding model, pushes chunks to Azure AI Search.
- QA Generation (run.qa_generation, azureml/query.py, promptflow/qa_generation) → fetches chunks, calls Azure OpenAI via llm module, outputs Q&A pairs.
- Querying & Reranking (run.querying, azureml/query.py, promptflow/querying) → executes various search types on Azure AI Search, optionally expands queries, reranks using llm/cross-encoder, collects enriched context.
- Evaluation (run.evaluation, azureml/eval.py, promptflow/evaluation) → computes metrics (text-distance, embedding similarities, LLM-judged), logs to MLFlow, builds reports & visualizations.
- Sampling (optional) → pre-cluster documents, select representative subset, feed into pipeline.
- Checkpointing (optional) → skip steps already done.

##### External Services & Infrastructure
- Azure AI Search Service (indexes + vector / semantic search)
- Azure Document Intelligence (prebuilt-layout or fallback)
- Azure OpenAI Service (embedding + chat completions)
- Azure ML (compute, MLFlow tracking, pipeline execution)
- Azure Storage (data / artifacts)
- Azure Key Vault (secret management)
- PromptFlow (serverless orchestration)
- Bicep/ARM templates (infra provisioning)

##### Architectural Patterns & Principles
- Modular library: separation of concerns (config, ingestion, embedding, IO, LLM, evaluation).
- Pipeline-driven orchestration: CLI scripts, Azure ML SDK pipelines, and PromptFlow DAGs reuse the same core library modules.
- Config-driven experimentation: use of combinatorial / sampling strategies in config.json.
- Checkpointing & idempotence for robust re-runs.
- Extensibility points: custom doc loaders, embedding factory, reranker interface, evaluation metrics plug-in.

##### Final Advice
- Asynchronous vs synchronous calls (e.g., document ingestion is I/O bound; LLM calls are external HTTP).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Orchestration
    DevEnv["Dev Environment<br/>Local / DevContainer"]:::compute
    subgraph "Orchestration" 
        direction TB
        subgraph "CLI" 
            direction TB
            IDX["01_index.py"]:::compute
            QA["02_qa_generation.py"]:::compute
            QRY["03_querying.py"]:::compute
            EVL["04_evaluation.py"]:::compute
        end
        subgraph "Azure ML Pipeline"
            direction TB
            AML_PIPE["pipeline.py"]:::compute
            AML_IDX["index.py"]:::compute
            AML_QRY["query.py"]:::compute
            AML_EVL["eval.py"]:::compute
        end
        subgraph "PromptFlow"
            direction TB
            PF_DAG["flow.dag.yaml"]:::compute
            PF_SETUP["setup_env.py"]:::compute
            PF_IDX["create_index.py"]:::compute
            PF_QA["generate_qa.py"]:::compute
            PF_QRY["querying.py"]:::compute
            PF_EVL["evaluation.py"]:::compute
            PF_WHL["custom_environment .whl"]:::compute
        end
    end

    %% Sampling branch
    Sampling["Sampling"]:::core
    DevEnv --> Sampling
    Sampling --> Ingest

    %% Core Library Modules
    subgraph "Core Library" 
        direction TB
        Config["Config Loader"]:::config
        Checkpoint["Checkpoint"]:::checkpoint
        Ingest["Document Ingestion"]:::core
        Embedding["Embedding"]:::core
        Indexing["Indexing"]:::core
        QA_Gen["QA Generation"]:::core
        Query_Rerank["Query & Rerank"]:::core
        Evaluation["Evaluation"]:::core
        IO["IO Abstractions"]:::core
    end

    %% Orchestration to core
    DevEnv --> CLI
    DevEnv --> AzureML
    DevEnv --> PromptFlow

    CLI --> Ingest
    AzureML --> Ingest
    PromptFlow --> Ingest

    %% Pipeline sequence
    Ingest --> Embedding
    Embedding --> Indexing
    Indexing --> QA_Gen
    QA_Gen --> Query_Rerank
    Query_Rerank --> Evaluation

    %% Cross cutting
    Config -.-> Ingest
    Config -.-> Embedding
    Config -.-> Indexing
    Config -.-> QA_Gen
    Config -.-> Query_Rerank
    Config -.-> Evaluation

    Checkpoint -.-> Ingest
    Checkpoint -.-> Embedding
    Checkpoint -.-> Indexing
    Checkpoint -.-> QA_Gen
    Checkpoint -.-> Query_Rerank
    Checkpoint -.-> Evaluation

    %% Data Stores
    LocalFS["Local File System"]:::data
    AzureBlob["Azure Storage"]:::data
    MLFlowStore["MLFlow Artifact Store"]:::data

    Ingest -->|raw docs| LocalFS
    Indexing -->|index data| AzureBlob
    Evaluation -->|reports & logs| MLFlowStore

    %% External Services
    AzureSearch["Azure AI Search"]:::external
    DocInt["Azure Document Intelligence"]:::external
    OpenAI["Azure OpenAI"]:::external
    AzureMLSvc["Azure ML Service"]:::external
    KeyVault["Azure Key Vault"]:::external

    Ingest -->|layout analysis| DocInt
    Embedding -->|generate embeddings| OpenAI
    Indexing -->|push chunks| AzureSearch
    QA_Gen -->|chat completions| OpenAI
    Query_Rerank -->|search queries| AzureSearch
    Evaluation -->|metrics logging| AzureMLSvc
    DevEnv -->|secrets| KeyVault

    %% Infra Provisioning
    subgraph "Infrastructure Provisioning" 
        direction TB
        BICEP_MAIN["main.bicep"]:::infra
        BICEP_PARAM["main.bicepparam"]:::infra
        NETWORK["network_isolation.bicep"]:::infra
        SEARCH_INFRA["search-services.bicep"]:::infra
        KV_INFRA["keyvault.bicep"]:::infra
        STORAGE_INFRA["storage.bicep"]:::infra
        ML_INFRA["machineLearning.bicep"]:::infra
    end
    DevEnv --> BICEP_MAIN
    BICEP_MAIN --> NETWORK
    BICEP_MAIN --> SEARCH_INFRA
    BICEP_MAIN --> KV_INFRA
    BICEP_MAIN --> STORAGE_INFRA
    BICEP_MAIN --> ML_INFRA

    %% Click Events
    click IDX "https://github.com/microsoft/rag-experiment-accelerator/blob/development//01_index.py"
    click QA "https://github.com/microsoft/rag-experiment-accelerator/blob/development//02_qa_generation.py"
    click QRY "https://github.com/microsoft/rag-experiment-accelerator/blob/development//03_querying.py"
    click EVL "https://github.com/microsoft/rag-experiment-accelerator/blob/development//04_evaluation.py"
    click AML_PIPE "https://github.com/microsoft/rag-experiment-accelerator/blob/development//azureml/pipeline.py"
    click AML_IDX "https://github.com/microsoft/rag-experiment-accelerator/blob/development//azureml/index.py"
    click AML_QRY "https://github.com/microsoft/rag-experiment-accelerator/blob/development//azureml/query.py"
    click AML_EVL "https://github.com/microsoft/rag-experiment-accelerator/blob/development//azureml/eval.py"
    click PF_DAG "https://github.com/microsoft/rag-experiment-accelerator/blob/development//promptflow/rag-experiment-accelerator/flow.dag.yaml"
    click PF_SETUP "https://github.com/microsoft/rag-experiment-accelerator/blob/development//promptflow/rag-experiment-accelerator/setup/setup_env.py"
    click PF_IDX "https://github.com/microsoft/rag-experiment-accelerator/blob/development//promptflow/rag-experiment-accelerator/index/create_index.py"
    click PF_QA "https://github.com/microsoft/rag-experiment-accelerator/blob/development//promptflow/rag-experiment-accelerator/qa_generation/generate_qa.py"
    click PF_QRY "https://github.com/microsoft/rag-experiment-accelerator/blob/development//promptflow/rag-experiment-accelerator/querying/querying.py"
    click PF_EVL "https://github.com/microsoft/rag-experiment-accelerator/blob/development//promptflow/rag-experiment-accelerator/evaluation/evaluation.py"
    click PF_WHL "https://github.com/microsoft/rag-experiment-accelerator/blob/development//promptflow/rag-experiment-accelerator/custom_environment/rag_experiment_accelerator-0.9-py3-none-any.whl"
    click Config "https://github.com/microsoft/rag-experiment-accelerator/tree/development//rag_experiment_accelerator/config/"
    click Ingest "https://github.com/microsoft/rag-experiment-accelerator/tree/development//rag_experiment_accelerator/doc_loader/"
    click Embedding "https://github.com/microsoft/rag-experiment-accelerator/tree/development//rag_experiment_accelerator/embedding/"
    click IO "https://github.com/microsoft/rag-experiment-accelerator/tree/development//rag_experiment_accelerator/io/"
    click Checkpoint "https://github.com/microsoft/rag-experiment-accelerator/tree/development//rag_experiment_accelerator/checkpoint/"
    click QA_Gen "https://github.com/microsoft/rag-experiment-accelerator/tree/development//rag_experiment_accelerator/llm/"
    click Query_Rerank "https://github.com/microsoft/rag-experiment-accelerator/blob/development//rag_experiment_accelerator/reranking/reranker.py"
    click Evaluation "https://github.com/microsoft/rag-experiment-accelerator/tree/development//rag_experiment_accelerator/evaluation/"
    click Sampling "https://github.com/microsoft/rag-experiment-accelerator/tree/development//rag_experiment_accelerator/sampling/"
    click BICEP_MAIN "https://github.com/microsoft/rag-experiment-accelerator/blob/development//infra/main.bicep"
    click BICEP_PARAM "https://github.com/microsoft/rag-experiment-accelerator/blob/development//infra/main.bicepparam"
    click NETWORK "https://github.com/microsoft/rag-experiment-accelerator/blob/development//infra/network/network_isolation.bicep"
    click SEARCH_INFRA "https://github.com/microsoft/rag-experiment-accelerator/blob/development//infra/shared/search-services.bicep"
    click KV_INFRA "https://github.com/microsoft/rag-experiment-accelerator/blob/development//infra/shared/keyvault.bicep"
    click STORAGE_INFRA "https://github.com/microsoft/rag-experiment-accelerator/blob/development//infra/shared/storage.bicep"
    click ML_INFRA "https://github.com/microsoft/rag-experiment-accelerator/blob/development//infra/shared/machineLearning.bicep"

    %% Styles
    classDef compute fill:#D0E6FF,stroke:#0284C7,color:#034467
    classDef core fill:#D6F6D6,stroke:#2E7D32,color:#1B5E20
    classDef data fill:#FFF9C4,stroke:#F9A825,color:#795548
    classDef external fill:#FFE0B2,stroke:#FB8C00,color:#E65100
    classDef infra fill:#E8EAF6,stroke:#3949AB,color:#1A237E
    classDef config fill:#F3E5F5,stroke:#8E24AA,color:#4A148C,shape:hexagon
    classDef checkpoint fill:#FFEBEE,stroke:#C62828,color:#8E0000,shape:hexagon
```
