# sbintuitions/flexeval architecture

> Architecture diagram of the GitHub repository sbintuitions/flexeval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sbintuitions/flexeval
- Repository: https://github.com/sbintuitions/flexeval

## Overview

##### Project Overview & Purpose
- FlexEval is a flexible evaluation tool for language models. It supports a wide range of evaluation tasks—from zero/few-shot in-context learning to open-ended text generation benchmarks, perplexity evaluation, and multiple-choice tasks.
- The modular design emphasizes extensibility and reproducibility. It leverages configuration files (often in JSONNET format) for specifying evaluation setups and integrates with various language model providers (Hugging Face, OpenAI, vLLM, etc.).

##### Main Components
- Core Engine (flexeval package):
- Evaluation Modules: Components that manage evaluation workflows such as evaluating chat responses, generation tasks, multiple-choice, and perplexity (e.g., flexeval/core/evaluate_*.py).
- Data Handling: Modules for managing different dataset types (chat_dataset, generation_dataset, text_dataset) along with support for tokenization (tokenizer submodule).
- Language Model Abstraction: Modules that interface with different language model APIs (HuggingFaceLM, OpenAI API, vLLM, etc.) under flexeval/core/language_model.
- Metrics: A variety of metric modules (BLEU, ROUGE, exact match, etc.) under flexeval/core/metric to compute and reconcile evaluation results.
- Few-shot & Prompt Components: Few-shot generator and prompt templating (flexeval/core/few_shot_generator and flexeval/core/prompt_template) for constructing evaluation prompts.
- Result Recording: Modules that record and store output results, including local recording and WandB integration.
- Reward Modeling: Additional evaluation components that support reward-based assessments.

- Configuration & Presets:
- Preset Configs: A directory (flexeval/preset_configs) containing pre-built evaluation setups for various tasks.
- Examples: Use-case examples (directory “examples”) illustrating how to configure and run evaluations.

- External Interfaces & Tools:
- Command-line Scripts: Utilities in flexeval/scripts that act as entry points (e.g., flexeval_lm, flexeval_pairwise) to launch evaluations.
- CI/CD and Build/Deployment: GitHub workflows (.github/workflows) and Dockerfile indicating containerization and continuous integration.
- Documentation: Comprehensive instructions and guides in the docs/ folder supporting users on setup, configuration, and usage.

##### Mapping Relationships and Data Flow
- The configuration then directs the evaluation pipeline:
- The evaluation driver instantiates the relevant dataset handler (e.g., chat_dataset or text_dataset) to load data.
- A few-shot generator or prompt template module processes the data to form prompts.
- The language model module abstracts callouts to external APIs or model interfaces, generating responses.
- The evaluation modules then compute metrics by comparing model outputs with ground truth using the metric submodules.
- The result recorder captures and writes out the evaluation outputs (config, metrics, outputs).
- External services: Model APIs (e.g., OpenAI, Hugging Face) and potential integrations with logging services (like WandB) are invoked via specialized modules.
- CI pipelines and containerization support deployment and repeated, reproducible evaluations.

##### Architectural Patterns & Design Principles
- Modular / Plug-in Architecture:
- Clear separation of concerns (data ingestion, model interfacing, evaluation, recording) allows developers to swap or extend any component.
- Config-Driven Execution:
- Extensive use of configuration files (JSONNET) and presets drives the behavior of various modules.
- Abstraction Layers:
- The language model module abstracts away differences between APIs, providing a uniform interface (polymorphism).
- Testability & Reproducibility:
- The project structure includes comprehensive tests (tests/ directory) and emphasizes reproducibility (saving configs and outputs).

##### Final Advice
- Encourage decomposing the system into as many clearly defined subcomponents as possible.
- Focus not only on the main data and control flows but also highlight extension points for adding new evaluation modules or language model integrations.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Subgraph: CLI & Entry
    subgraph "CLI & Entry"
        CLI["CLI Scripts"]:::cli
    end

    %% Subgraph: Configuration & Presets
    subgraph "Configuration & Presets"
        Config["Configuration Loader"]:::config
        Preset["Preset Configs"]:::config
        Examples["Examples"]:::config
    end

    %% Subgraph: Data & Preprocessing
    subgraph "Data & Preprocessing"
        ChatData["Chat Dataset"]:::data
        GenData["Generation Dataset"]:::data
        TextData["Text Dataset"]:::data
    end

    %% Subgraph: Prompt Generation
    subgraph "Prompt Generation"
        FewShot["Few-shot Generator"]:::prompt
        PromptTpl["Prompt Template"]:::prompt
    end

    %% Subgraph: Language Model Abstraction
    subgraph "Language Model Abstraction"
        LM_HF["HuggingFaceLM"]:::lm
        LM_OpenAI["OpenAI API"]:::lm
        LM_Litellm["LiteLLM API"]:::lm
        LM_OpenAIBatch["OpenAIBatch API"]:::lm
        LM_VLLM["vLLM Model"]:::lm
    end

    %% Subgraph: Evaluation Modules
    subgraph "Evaluation Modules"
        Eval_Chat["Evaluate Chat Response"]:::eval
        Eval_Generation["Evaluate Generation"]:::eval
        Eval_MC["Evaluate Multiple Choice"]:::eval
        Eval_Pairwise["Evaluate Pairwise"]:::eval
        Eval_Perplexity["Evaluate Perplexity"]:::eval
        Eval_Reward["Evaluate Reward Model"]:::eval
    end

    %% Separate subgraph for Reward Modeling components
    subgraph "Reward Modeling"
        RewardModel["Reward Modeling"]:::eval
    end

    %% Metrics Calculation
    Metrics["Metrics Calculation"]:::metric

    %% Subgraph: Result Recorder
    subgraph "Result Recorder"
        LocalRec["Local Recorder"]:::result
        WandBRec["WandB Recorder"]:::result
    end

    %% Subgraph: CI/CD & Deployment
    subgraph "CI/CD & Deployment"
        CI["GitHub Workflows"]:::deploy
        Docker["Dockerfile"]:::deploy
    end

    %% Documentation Node
    Doc["Documentation"]:::doc

    %% Subgraph: Additional CLI Scripts
    subgraph "Additional CLI Scripts"
        LM_Script["flexeval_lm.py"]:::cli
        Pairwise_Script["flexeval_pairwise.py"]:::cli
        Reward_Script["flexeval_reward.py"]:::cli
    end

    %% Main Flow Connections
    CLI -->|"loads"| Config
    Config -->|"uses"| Preset
    Config -->|"exemplified_by"| Examples
    Config -->|"feeds_data_to"| ChatData
    Config -->|"feeds_data_to"| GenData
    Config -->|"feeds_data_to"| TextData
    ChatData -->|"provides_data"| FewShot
    GenData -->|"provides_data"| FewShot
    TextData -->|"provides_data"| PromptTpl
    FewShot -->|"generates_prompt"| PromptTpl
    PromptTpl -->|"sends_prompt"| LM_HF
    PromptTpl -->|"sends_prompt"| LM_OpenAI
    PromptTpl -->|"sends_prompt"| LM_Litellm
    PromptTpl -->|"sends_prompt"| LM_OpenAIBatch
    PromptTpl -->|"sends_prompt"| LM_VLLM
    LM_HF -->|"returns_response"| Eval_Chat
    LM_OpenAI -->|"returns_response"| Eval_Generation
    LM_Litellm -->|"returns_response"| Eval_MC
    LM_OpenAIBatch -->|"returns_response"| Eval_Pairwise
    LM_VLLM -->|"returns_response"| Eval_Perplexity
    PromptTpl -->|"sends_prompt"| Eval_Reward
    Eval_Chat --> Metrics
    Eval_Generation --> Metrics
    Eval_MC --> Metrics
    Eval_Pairwise --> Metrics
    Eval_Perplexity --> Metrics
    Eval_Reward --> Metrics
    PromptTpl -->|"sends_prompt"| RewardModel
    RewardModel --> Metrics
    Metrics -->|"records_output"| LocalRec
    Metrics -->|"records_output"| WandBRec

    %% Connections to Additional CLI Scripts and External Tools
    CLI --- LM_Script
    CLI --- Pairwise_Script
    CLI --- Reward_Script
    CLI --- CI
    CLI --- Docker
    CLI --- Doc

    %% Click Events for Core Engine - Evaluation Modules
    click Eval_Chat "https://github.com/sbintuitions/flexeval/blob/main/flexeval/core/evaluate_chat_response.py"
    click Eval_Generation "https://github.com/sbintuitions/flexeval/blob/main/flexeval/core/evaluate_generation.py"
    click Eval_MC "https://github.com/sbintuitions/flexeval/blob/main/flexeval/core/evaluate_multiple_choice.py"
    click Eval_Pairwise "https://github.com/sbintuitions/flexeval/blob/main/flexeval/core/evaluate_pairwise.py"
    click Eval_Perplexity "https://github.com/sbintuitions/flexeval/blob/main/flexeval/core/evaluate_perplexity.py"
    click Eval_Reward "https://github.com/sbintuitions/flexeval/blob/main/flexeval/core/evaluate_reward_model.py"

    %% Click Events for Core Engine - Data Handling
    click ChatData "https://github.com/sbintuitions/flexeval/tree/main/flexeval/core/chat_dataset"
    click GenData "https://github.com/sbintuitions/flexeval/tree/main/flexeval/core/generation_dataset"
    click TextData "https://github.com/sbintuitions/flexeval/tree/main/flexeval/core/text_dataset"

    %% Click Events for Core Engine - Language Model Abstraction
    click LM_HF "https://github.com/sbintuitions/flexeval/blob/main/flexeval/core/language_model/hf_lm.py"
    click LM_OpenAI "https://github.com/sbintuitions/flexeval/blob/main/flexeval/core/language_model/openai_api.py"
    click LM_Litellm "https://github.com/sbintuitions/flexeval/blob/main/flexeval/core/language_model/litellm_api.py"
    click LM_OpenAIBatch "https://github.com/sbintuitions/flexeval/blob/main/flexeval/core/language_model/openai_batch_api.py"
    click LM_VLLM "https://github.com/sbintuitions/flexeval/blob/main/flexeval/core/language_model/vllm_model.py"

    %% Click Event for Core Engine - Metrics
    click Metrics "https://github.com/sbintuitions/flexeval/tree/main/flexeval/core/metric"

    %% Click Events for Core Engine - Few-shot & Prompt Components
    click FewShot "https://github.com/sbintuitions/flexeval/tree/main/flexeval/core/few_shot_generator"
    click PromptTpl "https://github.com/sbintuitions/flexeval/tree/main/flexeval/core/prompt_template"

    %% Click Events for Core Engine - Result Recording
    click LocalRec "https://github.com/sbintuitions/flexeval/blob/main/flexeval/core/result_recorder/local_recorder.py"
    click WandBRec "https://github.com/sbintuitions/flexeval/blob/main/flexeval/core/result_recorder/wandb_recorder.py"

    %% Click Events for Core Engine - Reward Modeling
    click RewardModel "https://github.com/sbintuitions/flexeval/tree/main/flexeval/core/reward_model"
    click RewardModel "https://github.com/sbintuitions/flexeval/tree/main/flexeval/core/reward_bench_dataset"

    %% Click Events for Configuration & Presets
    click Preset "https://github.com/sbintuitions/flexeval/tree/main/flexeval/preset_configs"
    click Examples "https://github.com/sbintuitions/flexeval/tree/main/examples/sarashina_2_2_evaluation"

    %% Click Events for External Interfaces - Command-line Scripts
    click LM_Script "https://github.com/sbintuitions/flexeval/blob/main/flexeval/scripts/flexeval_lm.py"
    click Pairwise_Script "https://github.com/sbintuitions/flexeval/blob/main/flexeval/scripts/flexeval_pairwise.py"
    click Reward_Script "https://github.com/sbintuitions/flexeval/blob/main/flexeval/scripts/flexeval_reward.py"

    %% Click Events for CI/CD and Deployment and Documentation
    click CI "https://github.com/sbintuitions/flexeval/tree/main/.github/workflows"
    click Docker "https://github.com/sbintuitions/flexeval/tree/main/Dockerfile"
    click Doc "https://github.com/sbintuitions/flexeval/tree/main/docs"

%% Class Definitions for Colors
classDef cli fill:#f1c40f,stroke:#000,stroke-width:2px;
classDef config fill:#e67e22,stroke:#000,stroke-width:2px;
classDef data fill:#2ecc71,stroke:#000,stroke-width:2px;
classDef prompt fill:#9b59b6,stroke:#000,stroke-width:2px;
classDef lm fill:#3498db,stroke:#000,stroke-width:2px;
classDef eval fill:#e74c3c,stroke:#000,stroke-width:2px;
classDef metric fill:#1abc9c,stroke:#000,stroke-width:2px;
classDef result fill:#95a5a6,stroke:#000,stroke-width:2px;
classDef deploy fill:#d35400,stroke:#000,stroke-width:2px;
classDef doc fill:#34495e,stroke:#000,stroke-width:2px;
```
