# rennykurian/deepeval architecture

> Architecture diagram of the GitHub repository rennykurian/deepeval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rennykurian/deepeval
- Repository: https://github.com/rennykurian/deepeval

## Overview

1. **Repository shape**
   This repo is primarily a **Python package** named `deepeval`, plus a separate **Next.js documentation site** under `docs/`, a large **test suite**, and example notebooks/scripts. The runtime product is the Python library; the docs site and tests are supporting systems around it.

2. **Core purpose**
   DeepEval is an **LLM evaluation framework**. Its central abstraction is: a user-defined application produces outputs, DeepEval wraps those outputs in **test cases**, then applies **metrics** to score quality, safety, faithfulness, tool use, multi-turn behavior, and benchmark performance.

3. **Main evaluation pipeline**
   The primary flow is: **test case or dataset → metric(s) → model-backed or deterministic scoring → result/explanation**. This flow appears in `deepeval/evaluate`, `deepeval/test_case`, `deepeval/dataset`, and `deepeval/metrics`. `assert_test()` and `evaluate()` are the high-level entry points, while `deepeval test run` exposes the same execution path through the CLI.

4. **Metrics subsystem**
   `deepeval/metrics` is the largest subsystem and is organized by use case: RAG, agentic, conversational, safety, multimodal, MCP, non-LLM, and custom LLM-judge metrics. Most metric packages include a **template**, often a **schema**, and a concrete metric class. Shared behavior lives in `base_metric.py`, `indicator.py`, and utility modules. Some metrics are deterministic; others call LLMs or embedding models through the model layer.

5. **Model abstraction layer**
   `deepeval/models` provides provider-neutral interfaces for both **LLMs** and **embedding models**. Provider wrappers include OpenAI, Anthropic, Azure, Gemini, Bedrock, Ollama, LiteLLM, OpenRouter, etc. The metrics and synthesizer modules depend on this layer so that the framework can evaluate with “any model” rather than a single vendor. Retry and provider utilities sit here as well.

6. **Tracing and observability**
   `deepeval/tracing` is the observability backbone. The `@observe` API, span/trace context, patchers, and OpenTelemetry exporter let DeepEval instrument LLM apps, agents, tools, and retrievers. Tracing supports both live capture and **offline evals** over collected traces. This is the key boundary between the user’s application code and DeepEval’s evaluation engine.

7. **Integration adapters**
   The `deepeval/integrations` tree contains framework-specific adapters for **OpenAI, OpenAI Agents, Anthropic, LangChain, LangGraph, CrewAI, LlamaIndex, Pydantic AI, Hugging Face, AWS AgentCore, and OpenInference**. These are not separate apps; they are glue layers that patch or callback into third-party runtimes and forward events into DeepEval tracing and metrics collection.

8. **Datasets, goldens, and test runs**
   `deepeval/dataset` and `deepeval/test_run` manage evaluation data: **goldens**, generated test cases, iteration over datasets, run metadata, hooks, caching, and hyperparameters. This layer is what turns raw examples into repeatable evaluation runs and is also used by tracing-based workflows to attach tests to spans.

9. **Synthetic data and simulation**
   `deepeval/synthesizer` generates goldens from documents, contexts, or existing test data. It includes document chunking, context generation, and prompt/template management. `deepeval/simulator` and `conversation_simulator` generate or replay multi-turn conversations. These subsystems feed the dataset layer rather than the metrics layer directly.

10. **Benchmarks**
    `deepeval/benchmarks` contains self-contained benchmark implementations such as MMLU, HellaSwag, GSM8K, BIG-Bench Hard, TruthfulQA, DROP, HumanEval, and others. Each benchmark has its own task/template/mode definitions and result schema. This subsystem is distinct from custom app evaluation: it evaluates models against standardized benchmark datasets.

11. **Prompt optimization**
    `deepeval/optimizer` is a separate optimization pipeline for improving prompts using evaluation feedback. It includes multiple algorithms such as **COPRO, GEPA, MIPROv2, and SIMBA**, plus a scorer/rewriter/policy layer. Architecturally, it consumes evaluation results and produces improved prompt variants, forming an outer optimization loop around the core eval engine.

12. **Confident AI platform boundary**
    `deepeval/confident`, `annotation`, and parts of `evaluate` and `dataset` connect to the **Confident AI** cloud platform. This is the optional persistence/sync boundary for logging datasets, traces, comparisons, and evaluation results. The README makes clear that the library works locally, but can also sync outcomes to the platform for sharing and storage.

13. **CLI and developer experience**
    `deepeval/cli` exposes user-facing commands for testing, generation, login, dotenv handling, and a local server entrypoint. This CLI is the operational front door for the repo’s primary workflows, especially running evaluations in a pytest-like way and managing environment setup.

14. **Repo support systems**
    The `docs/` app is a separate content-and-marketing surface built with Next.js, MDX, and custom components. The `tests/` tree is unusually comprehensive and includes core, metrics, tracing, integrations, docs validation, and snapshot-style schema fixtures; it serves as the main quality gate for this library. GitHub Actions workflows enforce formatting, core tests, integration tests, and changelog generation.

Main entry points: [`deepeval/metrics`](https://github.com/rennykurian/deepeval/tree/main/deepeval/metrics) (Metrics hub), [`deepeval/evaluate/evaluate.py`](https://github.com/rennykurian/deepeval/blob/main/deepeval/evaluate/evaluate.py) (Evaluate), [`deepeval/tracing`](https://github.com/rennykurian/deepeval/tree/main/deepeval/tracing) (Tracing), [`deepeval/dataset`](https://github.com/rennykurian/deepeval/tree/main/deepeval/dataset) (Datasets)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Core library"]
  node_entry["Public API<br/>top-level API<br/>[__init__.py]"]
  node_eval["Evaluate<br/>eval engine<br/>[evaluate.py]"]
  node_cases["Test cases<br/>case models"]
  node_dataset[("Datasets<br/>data layer")]
  node_metrics{{"Metrics hub<br/>metric system"}}
  node_ragm["RAG metrics<br/>metric family"]
  node_agentm["Agent metrics<br/>metric family<br/>[tool_use.py]"]
  node_convm["Conversation metrics<br/>metric family<br/>[turn_relevancy.py]"]
  node_safetym["Safety metrics<br/>metric family<br/>[toxicity.py]"]
  node_multi["Multimodal metrics<br/>metric family"]
  node_models[("Model layer<br/>provider abstraction")]
  node_tracing{{"Tracing<br/>observability core"}}
  node_cli["CLI<br/>command entrypoint<br/>[main.py]"]
  node_confident["Confident AI<br/>cloud boundary<br/>[api.py]"]
end

subgraph group_ext["Extensions"]
  node_synth["Synthesizer<br/>data generation<br/>[synthesizer.py]"]
  node_sim["Simulator<br/>conversation sim"]
  node_bench{{"Benchmarks<br/>benchmark suite"}}
  node_opt{{"Optimizer<br/>prompt optimization"}}
  node_integrations["Integrations<br/>adapters"]
end

subgraph group_support["Support surfaces"]
  node_docs["Docs site<br/>nextjs site<br/>[layout.tsx]"]
  node_tests["Test suite<br/>quality gate"]
  node_examples["Examples<br/>samples"]
end

node_entry -->|"exposes"| node_eval
node_entry -->|"exposes"| node_tracing
node_eval -->|"consumes"| node_cases
node_eval -->|"iterates"| node_dataset
node_eval -->|"scores with"| node_metrics
node_metrics -->|"uses"| node_models
node_metrics -->|"includes"| node_ragm
node_metrics -->|"includes"| node_agentm
node_metrics -->|"includes"| node_convm
node_metrics -->|"includes"| node_safetym
node_metrics -->|"includes"| node_multi
node_tracing -->|"feeds"| node_eval
node_tracing -->|"attaches"| node_dataset
node_tracing -->|"instrumented by"| node_integrations
node_cli -->|"invokes"| node_eval
node_cli -->|"invokes"| node_synth
node_synth -->|"produces"| node_dataset
node_sim -->|"produces"| node_dataset
node_bench -->|"evaluates with"| node_models
node_opt -->|"consumes"| node_metrics
node_opt -->|"uses"| node_models
node_confident -->|"syncs"| node_dataset
node_confident -->|"syncs"| node_tracing
node_docs -->|"documents"| node_entry
node_tests -->|"verifies"| node_eval
node_tests -->|"verifies"| node_metrics
node_tests -->|"verifies"| node_tracing
node_tests -->|"verifies"| node_integrations
node_examples -->|"demonstrates"| node_eval
node_examples -->|"demonstrates"| node_tracing

click node_entry "https://github.com/rennykurian/deepeval/blob/main/deepeval/__init__.py"
click node_eval "https://github.com/rennykurian/deepeval/blob/main/deepeval/evaluate/evaluate.py"
click node_cases "https://github.com/rennykurian/deepeval/tree/main/deepeval/test_case"
click node_dataset "https://github.com/rennykurian/deepeval/tree/main/deepeval/dataset"
click node_metrics "https://github.com/rennykurian/deepeval/tree/main/deepeval/metrics"
click node_ragm "https://github.com/rennykurian/deepeval/blob/main/deepeval/metrics/contextual_relevancy/contextual_relevancy.py"
click node_agentm "https://github.com/rennykurian/deepeval/blob/main/deepeval/metrics/tool_use/tool_use.py"
click node_convm "https://github.com/rennykurian/deepeval/blob/main/deepeval/metrics/turn_relevancy/turn_relevancy.py"
click node_safetym "https://github.com/rennykurian/deepeval/blob/main/deepeval/metrics/toxicity/toxicity.py"
click node_multi "https://github.com/rennykurian/deepeval/tree/main/deepeval/metrics/multimodal_metrics"
click node_models "https://github.com/rennykurian/deepeval/tree/main/deepeval/models"
click node_tracing "https://github.com/rennykurian/deepeval/tree/main/deepeval/tracing"
click node_cli "https://github.com/rennykurian/deepeval/blob/main/deepeval/cli/main.py"
click node_confident "https://github.com/rennykurian/deepeval/blob/main/deepeval/confident/api.py"
click node_synth "https://github.com/rennykurian/deepeval/blob/main/deepeval/synthesizer/synthesizer.py"
click node_sim "https://github.com/rennykurian/deepeval/blob/main/deepeval/simulator/conversation_simulator.py"
click node_bench "https://github.com/rennykurian/deepeval/tree/main/deepeval/benchmarks"
click node_opt "https://github.com/rennykurian/deepeval/blob/main/deepeval/optimizer/prompt_optimizer.py"
click node_integrations "https://github.com/rennykurian/deepeval/tree/main/deepeval/integrations"
click node_docs "https://github.com/rennykurian/deepeval/blob/main/docs/app/layout.tsx"
click node_tests "https://github.com/rennykurian/deepeval/tree/main/tests"
click node_examples "https://github.com/rennykurian/deepeval/tree/main/examples"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_entry,node_eval,node_cases,node_dataset,node_metrics,node_ragm,node_agentm,node_convm,node_safetym,node_multi,node_models,node_tracing,node_cli,node_confident toneBlue
class node_synth,node_sim,node_bench,node_opt,node_integrations toneAmber
class node_docs,node_tests,node_examples toneMint
```

## Components

### Core library

Primary Python package for evaluation, tracing, models, and orchestration.

- [Public API](https://github.com/rennykurian/deepeval/blob/main/deepeval/__init__.py): top-level API · Main user-facing entry points for evaluation, assertions, and tracing. · `deepeval/__init__.py`
- [Evaluate](https://github.com/rennykurian/deepeval/blob/main/deepeval/evaluate/evaluate.py): eval engine · Runs test cases or datasets through metrics and produces results. · `deepeval/evaluate/evaluate.py`
- [Test cases](https://github.com/rennykurian/deepeval/tree/main/deepeval/test_case): case models · Single-turn, conversational, arena, and MCP test case definitions. · `deepeval/test_case`
- [Datasets](https://github.com/rennykurian/deepeval/tree/main/deepeval/dataset): data layer · Goldens, dataset iteration, run metadata, hooks, caching, and run helpers. · `deepeval/dataset`
- [Metrics hub](https://github.com/rennykurian/deepeval/tree/main/deepeval/metrics): metric system · Central metric packages and shared metric base utilities. · `deepeval/metrics`
- [RAG metrics](https://github.com/rennykurian/deepeval/blob/main/deepeval/metrics/contextual_relevancy/contextual_relevancy.py): metric family · Answer, contextual, and faithfulness metrics for retrieval-augmented apps. · `deepeval/metrics/contextual_relevancy/contextual_relevancy.py`
- [Agent metrics](https://github.com/rennykurian/deepeval/blob/main/deepeval/metrics/tool_use/tool_use.py): metric family · Tool use, task completion, plan, role, and step-efficiency metrics. · `deepeval/metrics/tool_use/tool_use.py`
- [Conversation metrics](https://github.com/rennykurian/deepeval/blob/main/deepeval/metrics/turn_relevancy/turn_relevancy.py): metric family · Multi-turn relevance, completeness, retention, and turn-level scoring. · `deepeval/metrics/turn_relevancy/turn_relevancy.py`
- [Safety metrics](https://github.com/rennykurian/deepeval/blob/main/deepeval/metrics/toxicity/toxicity.py): metric family · Bias, toxicity, PII leakage, misuse, and other guardrail metrics. · `deepeval/metrics/toxicity/toxicity.py`
- [Multimodal metrics](https://github.com/rennykurian/deepeval/tree/main/deepeval/metrics/multimodal_metrics): metric family · Image and text-to-image evaluation metrics. · `deepeval/metrics/multimodal_metrics`
- [Model layer](https://github.com/rennykurian/deepeval/tree/main/deepeval/models): provider abstraction · Provider-neutral LLM and embedding interfaces plus vendor wrappers. · `deepeval/models`
- [Tracing](https://github.com/rennykurian/deepeval/tree/main/deepeval/tracing): observability core · Observe API, spans, traces, patchers, offline evals, and OpenTelemetry export. · `deepeval/tracing`
- [CLI](https://github.com/rennykurian/deepeval/blob/main/deepeval/cli/main.py): command entrypoint · User-facing commands for tests, generation, dotenv handling, login, and server workflows. · `deepeval/cli/main.py`
- [Confident AI](https://github.com/rennykurian/deepeval/blob/main/deepeval/confident/api.py): cloud boundary · Optional sync and persistence boundary for datasets, traces, comparisons, and eval results. · `deepeval/confident/api.py`

### Extensions

Optional subsystems that generate data, optimize prompts, or connect to external runtimes.

- [Synthesizer](https://github.com/rennykurian/deepeval/blob/main/deepeval/synthesizer/synthesizer.py): data generation · Generates goldens from docs, contexts, or existing examples. · `deepeval/synthesizer/synthesizer.py`
- [Simulator](https://github.com/rennykurian/deepeval/blob/main/deepeval/simulator/conversation_simulator.py): conversation sim · Multi-turn conversation generation, replay, and control loops. · `deepeval/simulator/conversation_simulator.py`
- [Benchmarks](https://github.com/rennykurian/deepeval/tree/main/deepeval/benchmarks): benchmark suite · Self-contained standardized benchmark implementations and result types. · `deepeval/benchmarks`
- [Optimizer](https://github.com/rennykurian/deepeval/blob/main/deepeval/optimizer/prompt_optimizer.py): prompt optimization · Algorithms and scoring/rewriting loop for improving prompts from eval feedback. · `deepeval/optimizer/prompt_optimizer.py`
- [Integrations](https://github.com/rennykurian/deepeval/tree/main/deepeval/integrations): adapters · Framework-specific adapters for tracing and metric collection across third-party runtimes. · `deepeval/integrations`

### Support surfaces

Docs, tests, and examples around the core package.

- [Docs site](https://github.com/rennykurian/deepeval/blob/main/docs/app/layout.tsx): nextjs site · Separate Next.js documentation and marketing surface. · `docs/app/layout.tsx`
- [Test suite](https://github.com/rennykurian/deepeval/tree/main/tests): quality gate · Core, metrics, tracing, integrations, docs, and platform validation tests. · `tests`
- [Examples](https://github.com/rennykurian/deepeval/tree/main/examples): samples · Scripts, notebooks, and runnable examples demonstrating core workflows. · `examples`

## Connections

- Public API → Evaluate: exposes · Public API routes users into evaluation entry points.
- Public API → Tracing: exposes · Public API also exposes tracing and observation helpers.
- Evaluate → Test cases: consumes · Evaluation executes test cases and their variants.
- Evaluate → Datasets: iterates · Evaluation can run over datasets and goldens.
- Evaluate → Metrics hub: scores with · Evaluation delegates scoring to one or more metrics.
- Metrics hub → Model layer: uses · Model-backed metrics call into the provider-neutral model layer.
- Metrics hub → RAG metrics: includes · RAG-oriented metrics form one major metric family.
- Metrics hub → Agent metrics: includes · Agentic and tool-use metrics form another major family.
- Metrics hub → Conversation metrics: includes · Conversational and turn-level metrics form a separate family.
- Metrics hub → Safety metrics: includes · Safety and policy metrics are grouped here as well.
- Metrics hub → Multimodal metrics: includes · Image and multimodal metrics live under the metrics hub.
- Tracing → Evaluate: feeds · Offline evals and trace-derived workflows flow back into evaluation.
- Tracing → Datasets: attaches · Traces can be associated with datasets, goldens, and run metadata.
- Tracing → Integrations: instrumented by · Adapters patch third-party runtimes into the tracing backbone.
- CLI → Evaluate: invokes · CLI test commands invoke the same evaluation path as the library API.
- CLI → Synthesizer: invokes · CLI also exposes generation workflows.
- Synthesizer → Datasets: produces · Synthetic generation outputs goldens and dataset material.
- Simulator → Datasets: produces · Simulations generate or replay conversation data for datasets.
- Benchmarks → Model layer: evaluates with · Benchmarks run models against standardized tasks and datasets.
- Optimizer → Metrics hub: consumes · Prompt optimization uses evaluation results and metric signals.
- Optimizer → Model layer: uses · Optimization loops may call models for scoring and rewriting.
- Confident AI → Datasets: syncs · Cloud boundary persists or syncs datasets and related artifacts.
- Confident AI → Tracing: syncs · Cloud boundary also receives traces and evaluation outcomes.
- Docs site → Public API: documents · Docs explain the library’s primary APIs and workflows.
- Test suite → Evaluate: verifies · Core tests exercise the evaluation engine and its execution modes.
- Test suite → Metrics hub: verifies · Metric correctness is heavily validated by the suite.
- Test suite → Tracing: verifies · Tracing and observability behavior is covered extensively.
- Test suite → Integrations: verifies · Integration adapters are tested against recorded schemas and apps.
- Examples → Evaluate: demonstrates · Examples show library evaluation usage across common scenarios.
- Examples → Tracing: demonstrates · Examples also demonstrate tracing and observed applications.