# aws/fmeval architecture

> Architecture diagram of the GitHub repository aws/fmeval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aws/fmeval
- Repository: https://github.com/aws/fmeval

## Overview

##### Project type and purpose
- Project type: A Python library (not a web or mobile app) for evaluating foundation (large) language models.
- Purpose: Provide a modular framework for running evaluation “algorithms” (accuracy, toxicity, semantic robustness, prompt stereotyping) against any LLM by abstracting model invocation (ModelRunner interface), data loading, transforms pipelines, and reporting.

##### File structure
- Top-level:
– .github/, devtool (CI/tooling)
– examples/ (Jupyter notebooks demonstrating usage)
– src/fmeval/ (library code)
– test/ (unit & integration tests)
- Under src/fmeval/:
– data_loaders/: ingest built-in or custom datasets (JSONL, etc.)
– eval_algorithms/: one folder per metric/task implements EvalAlgorithm strategy
– model_runners/: strategy objects encapsulating calls to AWS Bedrock, SageMaker JumpStart, user-provided runners
– transforms/: pipeline pattern – chain of record-level transforms
– reporting/: assemble results into tables/cells, output formatting
- Config/build: pyproject.toml, setup.cfg, devtool scripts

##### README for additional insights
- Architecture pattern: strategy for ModelRunner, pipeline for Transform/TransformPipeline
- Telemetry: hooks in ModelRunner constructors to record usage
- Extension points: you add new ModelRunners or new Transforms/EvalAlgorithms

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Entry Point
    User["User / CLI / Notebook"]:::external
    Eval["Eval (src/fmeval/eval.py)"]:::core
    click Eval "https://github.com/aws/fmeval/blob/main/src/fmeval/eval.py"

    %% Core Modules
    subgraph "Core Library (src/fmeval)" 
        direction TB
        DL["Data Loaders"]:::core
        click DL "https://github.com/aws/fmeval/tree/main/src/fmeval/data_loaders/"
        EA["EvalAlgorithm Manager"]:::core
        click EA "https://github.com/aws/fmeval/blob/main/src/fmeval/eval_algo_mapping.py"
        subgraph "EvalAlgorithms" 
            direction TB
            BaseEA["Base EvalAlgorithm"]:::core
            click BaseEA "https://github.com/aws/fmeval/blob/main/src/fmeval/eval_algorithms/eval_algorithm.py"
            AccEA["classification_accuracy.py"]:::core
            FactEA["factual_knowledge.py"]:::core
            PromptEA["prompt_stereotyping.py"]:::core
            RobustEA["general_semantic_robustness.py"]:::core
            ToxicEA["toxicity.py"]:::core
            click AccEA "https://github.com/aws/fmeval/blob/main/src/fmeval/eval_algorithms/classification_accuracy.py"
            click FactEA "https://github.com/aws/fmeval/blob/main/src/fmeval/eval_algorithms/factual_knowledge.py"
            click PromptEA "https://github.com/aws/fmeval/blob/main/src/fmeval/eval_algorithms/prompt_stereotyping.py"
            click RobustEA "https://github.com/aws/fmeval/blob/main/src/fmeval/eval_algorithms/general_semantic_robustness.py"
            click ToxicEA "https://github.com/aws/fmeval/blob/main/src/fmeval/eval_algorithms/toxicity.py"
        end
        TP["TransformPipeline"]:::core
        click TP "https://github.com/aws/fmeval/blob/main/src/fmeval/transforms/transform_pipeline.py"
        subgraph "Transforms"
            direction TB
            TransBase["Transform / common"]:::core
            click TransBase "https://github.com/aws/fmeval/blob/main/src/fmeval/transforms/transform.py"
            click TransBase "https://github.com/aws/fmeval/blob/main/src/fmeval/transforms/common.py"
            MetricT["semantic_robustness_metrics.py"]:::core
            click MetricT "https://github.com/aws/fmeval/blob/main/src/fmeval/transforms/semantic_robustness_metrics.py"
        end
        subgraph "ModelRunners"
            direction TB
            MRInt["ModelRunner Interface"]:::core
            click MRInt "https://github.com/aws/fmeval/blob/main/src/fmeval/model_runners/model_runner.py"
            Bedrock["BedrockModelRunner"]:::plugin
            click Bedrock "https://github.com/aws/fmeval/blob/main/src/fmeval/model_runners/bedrock_model_runner.py"
            SMJS["SMJumpStartRunner"]:::plugin
            click SMJS "https://github.com/aws/fmeval/blob/main/src/fmeval/model_runners/sm_jumpstart_model_runner.py"
            SMEP["SMEndpointRunner"]:::plugin
            click SMEP "https://github.com/aws/fmeval/blob/main/src/fmeval/model_runners/sm_model_runner.py"
            subgraph "Composers" 
                direction TB
                Comp["composers.py"]:::core
                click Comp "https://github.com/aws/fmeval/blob/main/src/fmeval/model_runners/composers/composers.py"
            end
            subgraph "Extractors"
                direction TB
                Ext["extractor.py"]:::core
                click Ext "https://github.com/aws/fmeval/blob/main/src/fmeval/model_runners/extractors/extractor.py"
            end
        end
        Rep["Reporting"]:::core
        click Rep "https://github.com/aws/fmeval/tree/main/src/fmeval/reporting/"
        Telemetry["Telemetry Hooks"]:::external
        click Telemetry "https://github.com/aws/fmeval/blob/main/src/fmeval/perf_util.py"
    end

    %% Consumers
    Examples["Examples (notebooks)"]:::external
    click Examples "https://github.com/aws/fmeval/tree/main/examples/"

    Tests["Tests"]:::external
    click Tests "https://github.com/aws/fmeval/tree/main/test/"

    %% External Services
    subgraph "External LLM Services"
        direction TB
        AWSBed["AWS Bedrock"]:::external
        SMDest["SageMaker Endpoint"]:::external
        SMJump["SageMaker JumpStart"]:::external
    end

    %% Flows
    User -->|invokes| Eval
    Eval -->|loads data| DL
    DL -->|provides records| EA
    EA -->|builds pipeline| TP
    TP -->|calls predict| MRInt
    MRInt -->|to| AWSBed
    MRInt -->|to| SMDest
    MRInt -->|to| SMJump
    MRInt -->|logs| Telemetry
    TP -->|transforms| EA
    EA -->|sends results| Rep
    Rep -->|outputs| User
    User --> Examples
    User --> Tests

    %% Styles
    classDef core fill:#D0E8FF,stroke:#0366d6,color:#0366d6;
    classDef plugin fill:#DFFFE0,stroke:#28a745,color:#28a745;
    classDef external fill:#FFE0B2,stroke:#d97706,color:#d97706;
```
