# auto-playground/ragrank architecture

> Architecture diagram of the GitHub repository auto-playground/ragrank, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/auto-playground/ragrank
- Repository: https://github.com/auto-playground/ragrank

## Overview

##### Main Components

a) Core Components:
- Dataset Management System
- Evaluation Engine
- Metrics System
- LLM Integration Layer
- Prompt Management System

b) Integration Components:
- LangChain Integration
- LlamaIndex Integration
- OpenAI Integration

**┌─ Application Layer ─────────────────────────────┐**

- ┌─ Evaluation Engine ─┐                    │

- ┘                    │

- Core Services Layer ──────────────────────────┤

- ┌─ Metrics ─┐  ┌─ Dataset ─┐  ┌─ Prompt ─┐   │

- ┘  └───────────┘  └──────────┘   │

- Integration Layer ───────────────────────────┤

- ┌─ LLM Wrappers ────────────────────────┐   │

- OpenAI  LangChain  LlamaIndex        │   │

- ┘   │

- ┘

##### Component Details

- Rectangles with rounded corners for core services
- Hexagons for integration points
- Diamonds for evaluation components

##### Key Relationships to Highlight

a) Evaluation Flow:
- Dataset → Evaluation Engine → Metrics → Results
- LLM Integration → Evaluation Engine
- Prompts → LLM Integration

b) Integration Points:
- How external LLM services connect through the integration layer
- Demonstrate metric customization flow

##### Important Details

a) Metric Types (group them):
- Context-related metrics
- Response-related metrics
- Custom metrics

b) Dataset Handling:
- Data ingestion flow
- Validation and processing steps

##### Implementation Notes

- The modular nature of the system
- The extensibility of metrics and LLM integrations
- The evaluation pipeline flow
- Clear separation of concerns between components

##### Technical Details to Highlight

- Support for multiple LLM providers
- Custom metric creation capability
- Dataset format handling
- Integration with popular frameworks

Remember to:
- Version compatibility information
- Clear boundaries between components
- Public interfaces and extension points

This system design should provide a clear understanding of Ragrank's architecture while maintaining the right balance between detail and clarity.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Application
        EvalEngine["Evaluation Engine"]:::core
        click EvalEngine "https://github.com/Auto-Playground/ragrank/blob/main/src/ragrank/evaluation/base.py"
        EvalOutputs["Evaluation Outputs"]:::core
        click EvalOutputs "https://github.com/Auto-Playground/ragrank/blob/main/src/ragrank/evaluation/outputs.py"
    end

    subgraph Core_Services
        DatasetMgmt["Dataset Management"]:::core
        click DatasetMgmt "https://github.com/Auto-Playground/ragrank/tree/main/src/ragrank/dataset/"
        DatasetBase["Dataset Base"]:::core
        click DatasetBase "https://github.com/Auto-Playground/ragrank/blob/main/src/ragrank/dataset/base.py"
        DatasetReader["Dataset Reader"]:::core
        click DatasetReader "https://github.com/Auto-Playground/ragrank/blob/main/src/ragrank/dataset/reader.py"

        PromptMgmt["Prompt Management"]:::core
        click PromptMgmt "https://github.com/Auto-Playground/ragrank/tree/main/src/ragrank/prompt/"
        PromptBase["Prompt Base"]:::core
        click PromptBase "https://github.com/Auto-Playground/ragrank/blob/main/src/ragrank/prompt/base.py"
        PromptTemplates["Prompt Templates"]:::core
        click PromptTemplates "https://github.com/Auto-Playground/ragrank/blob/main/src/ragrank/prompt/_prompts.py"

        MetricsSystem["Metrics System"]:::core
        click MetricsSystem "https://github.com/Auto-Playground/ragrank/tree/main/src/ragrank/metric/"
        MetricBase["Metric Base"]:::core
        click MetricBase "https://github.com/Auto-Playground/ragrank/blob/main/src/ragrank/metric/base.py"

        subgraph Metrics
            ContextMetrics["Context Metrics"]:::metric
            click ContextMetrics "https://github.com/Auto-Playground/ragrank/tree/main/src/ragrank/metric/_context_related/"
            ResponseMetrics["Response Metrics"]:::metric
            click ResponseMetrics "https://github.com/Auto-Playground/ragrank/tree/main/src/ragrank/metric/_response_related/"
            CustomMetrics["Custom Metrics"]:::metric
            click CustomMetrics "https://github.com/Auto-Playground/ragrank/tree/main/src/ragrank/metric/_custom/"
        end
    end

    subgraph Integration_Layer
        LLMBase["LLM Base Interface"]:::integration
        click LLMBase "https://github.com/Auto-Playground/ragrank/blob/main/src/ragrank/llm/base.py"
        LLMUtils["LLM Utilities"]:::integration
        click LLMUtils "https://github.com/Auto-Playground/ragrank/blob/main/src/ragrank/utils/llm.py"

        subgraph LLM_Integrations
            OpenAI["OpenAI Integration"]:::llm
            click OpenAI "https://github.com/Auto-Playground/ragrank/blob/main/src/ragrank/integrations/openai/openai_llm.py"
            LangChain["LangChain Integration"]:::llm
            click LangChain "https://github.com/Auto-Playground/ragrank/blob/main/src/ragrank/integrations/langchain/langchain_llm_wrapper.py"
            LlamaIndex["LlamaIndex Integration"]:::llm
            click LlamaIndex "https://github.com/Auto-Playground/ragrank/blob/main/src/ragrank/integrations/llama_index/llamaindex_llm_wrapper.py"
        end
    end

    %% Relationships
    DatasetMgmt --> DatasetBase
    DatasetMgmt --> DatasetReader
    DatasetMgmt --> EvalEngine

    PromptMgmt --> PromptBase
    PromptMgmt --> PromptTemplates
    PromptMgmt --> LLMBase

    MetricsSystem --> MetricBase
    MetricBase --> ContextMetrics
    MetricBase --> ResponseMetrics
    MetricBase --> CustomMetrics

    LLMBase --> LLMUtils
    LLMBase --> OpenAI
    LLMBase --> LangChain
    LLMBase --> LlamaIndex

    EvalEngine --> EvalOutputs
    EvalEngine --> MetricsSystem
    EvalEngine --> LLMBase
    EvalEngine --> PromptMgmt

    %% Styles
    classDef core fill:#2374ab,stroke:#2374ab,color:#fff
    classDef integration fill:#48a9a6,stroke:#48a9a6,color:#fff
    classDef metric fill:#e4572e,stroke:#e4572e,color:#fff
    classDef llm fill:#17bebb,stroke:#17bebb,color:#fff

    %% Legend
    subgraph Legend
        CoreComp["Core Component"]:::core
        IntegrationComp["Integration Component"]:::integration
        MetricComp["Metric Component"]:::metric
        LLMComp["LLM Integration"]:::llm
    end
```
