# hustyichi/dify-eval architecture

> Architecture diagram of the GitHub repository hustyichi/dify-eval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hustyichi/dify-eval
- Repository: https://github.com/hustyichi/dify-eval

## Overview

##### Main Components
- Data Ingestion & Dataset Management:
- Modules under the “dify_eval/dataset” directory. This includes the dataset construction (dataset.py) and extraction submodules (extractor, csv_extractor, extractor_base.py). These components are responsible for reading local CSV files (e.g., example.csv) and creating the data sets.
- Evaluation Engine:
- Evaluation components (located in “dify_eval/evaluation”). This includes the evaluation routine, metrics, constants, and integration with RAGAS models. The DEFAULT_METRICS defined in evaluate.py drive automated scoring.
- Generation / Interaction Module:
- Generation modules (in “dify_eval/generation”), which include dify_chat.py and generation.py. This component likely handles requests to Dify for generating responses based on the uploaded datasets.
- External Service Integrations:
- Dify: The system sends tasks or data to the Dify service.
- Langfuse: The system uploads the prepared dataset to Langfuse for tracking and monitoring.
- Ragas / OpenAI path: For advanced evaluation using vector models and large model scoring. (Note that the README mentions alternative backends like Xinference.)

##### System Entry Points
- run.py:
- Initiates the workflow by uploading local datasets to Langfuse and submitting tasks to Dify.
- evaluate.py:
- Executes the evaluation of the data sets by invoking evaluation routines and metrics processing.
- build.py:
- Provides the logic for building or modifying the test dataset.

##### Relationships & Interactions
- Data Flow:
- The process starts with local CSV files (example.csv) and potentially built datasets (using build.py).
- The ingestion module uses extractors to parse and structure data.
- The processed data is then submitted to Langfuse via run.py, which in turn triggers task submission to Dify.
- Evaluation Flow:
- Post submission, the evaluation module kicks in with evaluate.py, using the metrics provided (and DEFAULT_METRICS from evaluate.py) to score responses and generate comparative analytics.
- Service Interactions:

##### Architectural Patterns & Design Principles
- Modularity and Separation of Concerns:
- Notice that the project is divided into clear modules (dataset, evaluation, generation) which implies a design that separates data handling, evaluation logic, and service communication.
- Pipeline Orchestration:
- The overall workflow can be seen as a pipeline: ingest data → extract and process → submit tasks → evaluate responses.
- Extensibility:

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Entry Points
    subgraph "Entry Points"
        runpy["run.py"]:::entry
        evaluatepy["evaluate.py"]:::entry
        buildpy["build.py"]:::entry
    end

    %% Data Input
    LocalCSV["Local CSV Files (example.csv)"]:::dataset

    %% Dataset Management
    subgraph "Dataset Management"
        DS["Dataset Construction"]:::dataset
        CSVExt["CSV Extractor"]:::dataset
        Ext["Extractor"]:::dataset
        ExtBase["Extractor Base"]:::dataset
    end

    %% Evaluation Engine
    subgraph "Evaluation Engine"
        EvalRoutine["Evaluation Routine"]:::evaluation
        Metrics["Metrics"]:::evaluation
        Constants["Constants"]:::evaluation
        RAGAS["RAGAS Integration"]:::evaluation
    end

    %% Generation Module
    subgraph "Generation Module"
        DifyChat["Dify Chat"]:::generation
        Generation["Generation"]:::generation
    end

    %% External Services
    subgraph "External Services"
        Dify["Dify<br>Large model service"]:::external
        Langfuse["Langfuse<br>Metrics tracking"]:::external
        RagasService["RAGAS/Model Scoring"]:::external
    end

    %% Data Flow Connections
    LocalCSV --> DS
    buildpy --> DS
    DS --> CSVExt
    DS --> Ext
    DS --> ExtBase
    DS --> runpy
    runpy --> Langfuse
    runpy --> Dify
    runpy --> DifyChat
    runpy --> Generation
    DifyChat --> Dify
    Generation --> Dify
    evaluatepy --> EvalRoutine
    EvalRoutine --> Metrics
    EvalRoutine --> Constants
    EvalRoutine --> RAGAS
    RAGAS --> RagasService

    %% Click Events
    click DS "https://github.com/hustyichi/dify-eval/blob/main/dify_eval/dataset/dataset.py"
    click CSVExt "https://github.com/hustyichi/dify-eval/blob/main/dify_eval/dataset/extractor/csv_extractor.py"
    click Ext "https://github.com/hustyichi/dify-eval/blob/main/dify_eval/dataset/extractor/extractor.py"
    click ExtBase "https://github.com/hustyichi/dify-eval/blob/main/dify_eval/dataset/extractor/extractor_base.py"
    click EvalRoutine "https://github.com/hustyichi/dify-eval/blob/main/dify_eval/evaluation/evaluation.py"
    click Metrics "https://github.com/hustyichi/dify-eval/blob/main/dify_eval/evaluation/metrics.py"
    click Constants "https://github.com/hustyichi/dify-eval/blob/main/dify_eval/evaluation/constants.py"
    click RAGAS "https://github.com/hustyichi/dify-eval/blob/main/dify_eval/evaluation/ragas_models.py"
    click DifyChat "https://github.com/hustyichi/dify-eval/blob/main/dify_eval/generation/dify_chat.py"
    click Generation "https://github.com/hustyichi/dify-eval/blob/main/dify_eval/generation/generation.py"
    click runpy "https://github.com/hustyichi/dify-eval/blob/main/run.py"
    click evaluatepy "https://github.com/hustyichi/dify-eval/blob/main/evaluate.py"
    click buildpy "https://github.com/hustyichi/dify-eval/blob/main/build.py"

    %% Styles
    classDef dataset fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef evaluation fill:#A9DFBF,stroke:#1D8348,stroke-width:2px;
    classDef generation fill:#D2B4DE,stroke:#6C3483,stroke-width:2px;
    classDef entry fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef external fill:#F5CBA7,stroke:#AF601A,stroke-width:2px;
```
