How microsoft/rag-experiment-accelerator works
Project Type & Purpose
• This is an open-source experimentation accelerator for Retrieval-Augmented Generation (RAG) built in Python.
• It provides both CLI scripts (01_index.py → 04_evaluation.py) and orchestrated pipelines (Azure ML, PromptFlow) to ingest documents, build/search indexes, generate Q&A, rerank results, and compute metrics.
• It integrates with Azure AI Search, Azure Document Intelligence, Azure OpenAI, Azure Machine Learning, MLFlow, and local data.
File Structure Analysis
• Top-level scripts: 01_index.py, 02_qa_generation.py, 03_querying.py, 04_evaluation.py (CLI entrypoints).
• azureml/: Azure ML SDK pipeline definitions and helper scripts.
• promptflow/: PromptFlow DAGs and custom environments to orchestrate the same four steps in a serverless fashion.
• infra/: Bicep/ARM templates to provision Azure resources (Search, Key Vault, ML, Storage, Cognitive Services).
• rag_experiment_accelerator/: Core Python library organized into submodules:
• config (load/validate experiment settings)
• doc_loader (multi-format document ingestion & chunking)
• embedding (create vector embeddings via Azure OAI or sentence-transformers)
• io (local file read/write abstractions & checkpointing)
• llm (prompt construction and response generation)
• reranking (LLM or cross-encoder based reranking)
• evaluation (end-to-end & component metrics, plotting, MLFlow integration)
• sampling (content-driven clustering & selection)
• checkpoint (skip reprocessing of completed steps)
• data/: Sample documents; artifacts/ for sampling outputs.
• config.sample.json / .env.template: config and environment variable blueprints.
Core Components & Interactions
• Development Environment (local or DevContainer) → orchestrates either CLI scripts, Azure ML pipeline, or PromptFlow flows.
• Document Ingestion (doc_loader) → produces chunks.
• Indexing (rag_experiment_accelerator.run.index & azureml/index.py & promptflow/index) → uses embedding model, pushes chunks to Azure AI Search.
• QA Generation (run.qa_generation, azureml/query.py, promptflow/qa_generation) → fetches chunks, calls Azure OpenAI via llm module, outputs Q&A pairs.
• Querying & Reranking (run.querying, azureml/query.py, promptflow/querying) → executes various search types on Azure AI Search, optionally expands queries, reranks using llm/cross-encoder, collects enriched context.
• Evaluation (run.evaluation, azureml/eval.py, promptflow/evaluation) → computes metrics (text-distance, embedding similarities, LLM-judged), logs to MLFlow, builds reports & visualizations.
• Sampling (optional) → pre-cluster documents, select representative subset, feed into pipeline.
• Checkpointing (optional) → skip steps already done.
External Services & Infrastructure
• Azure AI Search Service (indexes + vector / semantic search)
• Azure Document Intelligence (prebuilt-layout or fallback)
• Azure OpenAI Service (embedding + chat completions)
• Azure ML (compute, MLFlow tracking, pipeline execution)
• Azure Storage (data / artifacts)
• Azure Key Vault (secret management)
• PromptFlow (serverless orchestration)
• Bicep/ARM templates (infra provisioning)
Architectural Patterns & Principles
• Modular library: separation of concerns (config, ingestion, embedding, IO, LLM, evaluation).
• Pipeline-driven orchestration: CLI scripts, Azure ML SDK pipelines, and PromptFlow DAGs reuse the same core library modules.
• Config-driven experimentation: use of combinatorial / sampling strategies in config.json.
• Checkpointing & idempotence for robust re-runs.
• Extensibility points: custom doc loaders, embedding factory, reranker interface, evaluation metrics plug-in.
Final Advice
• Asynchronous vs synchronous calls (e.g., document ingestion is I/O bound; LLM calls are external HTTP).