How hustyichi/dify-eval works
Main Components
• Data Ingestion & Dataset Management:
• Modules under the “dify_eval/dataset” directory. This includes the dataset construction (dataset.py) and extraction submodules (extractor, csv_extractor, extractor_base.py). These components are responsible for reading local CSV files (e.g., example.csv) and creating the data sets.
• Evaluation Engine:
• Evaluation components (located in “dify_eval/evaluation”). This includes the evaluation routine, metrics, constants, and integration with RAGAS models. The DEFAULT_METRICS defined in evaluate.py drive automated scoring.
• Generation / Interaction Module:
• Generation modules (in “dify_eval/generation”), which include dify_chat.py and generation.py. This component likely handles requests to Dify for generating responses based on the uploaded datasets.
• External Service Integrations:
• Dify: The system sends tasks or data to the Dify service.
• Langfuse: The system uploads the prepared dataset to Langfuse for tracking and monitoring.
• Ragas / OpenAI path: For advanced evaluation using vector models and large model scoring. (Note that the README mentions alternative backends like Xinference.)
System Entry Points
• run.py:
• Initiates the workflow by uploading local datasets to Langfuse and submitting tasks to Dify.
• evaluate.py:
• Executes the evaluation of the data sets by invoking evaluation routines and metrics processing.
• build.py:
• Provides the logic for building or modifying the test dataset.
Relationships & Interactions
• Data Flow:
• The process starts with local CSV files (example.csv) and potentially built datasets (using build.py).
• The ingestion module uses extractors to parse and structure data.
• The processed data is then submitted to Langfuse via run.py, which in turn triggers task submission to Dify.
• Evaluation Flow:
• Post submission, the evaluation module kicks in with evaluate.py, using the metrics provided (and DEFAULT_METRICS from evaluate.py) to score responses and generate comparative analytics.
• Service Interactions:
Architectural Patterns & Design Principles
• Modularity and Separation of Concerns:
• Notice that the project is divided into clear modules (dataset, evaluation, generation) which implies a design that separates data handling, evaluation logic, and service communication.
• Pipeline Orchestration:
• The overall workflow can be seen as a pipeline: ingest data → extract and process → submit tasks → evaluate responses.
• Extensibility: