# confident-ai/deepeval architecture

> Architecture diagram of the GitHub repository confident-ai/deepeval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/confident-ai/deepeval
- Repository: https://github.com/confident-ai/deepeval

## Overview

DeepEval is a Python LLM evaluation framework for testing model applications, agent trajectories, and individual agent steps. Users define test cases or datasets, select metrics, and run evaluations to judge outputs and report results; models and local NLP back those judgments. Separate capabilities cover trace capture and inspection, synthetic data generation, conversation simulation, prompt optimization, and integrations with agent frameworks. The graph preserves these README-documented branches. Relationships are documented at a high level; unsampled implementation details and call paths are not inferred.

Main entry points: [`deepeval/evaluate/evaluate.py`](https://github.com/confident-ai/deepeval/blob/main/deepeval/evaluate/evaluate.py) (Evaluation runner), [`deepeval/test_case/llm_test_case.py`](https://github.com/confident-ai/deepeval/blob/main/deepeval/test_case/llm_test_case.py) (Test cases), [`deepeval/dataset/dataset.py`](https://github.com/confident-ai/deepeval/blob/main/deepeval/dataset/dataset.py) (Datasets), [`deepeval/metrics`](https://github.com/confident-ai/deepeval/tree/main/deepeval/metrics) (Evaluation metrics)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_evaluation["Evaluation workflow"]
  node_cases["Test cases<br/>[llm_test_case.py]"]
  node_datasets["Datasets<br/>[dataset.py]"]
  node_evaluate["Evaluation runner<br/>[evaluate.py]"]
end

subgraph group_judging["Judging and models"]
  node_metrics["Evaluation metrics"]
  node_models["Judge models"]
  node_classifiers["Safety classifiers"]
end

subgraph group_observability["Tracing and inspection"]
  node_trace["Tracing<br/>[tracing.py]"]
  node_inspect["Trace inspector<br/>[loader.py]"]
end

subgraph group_development["Data and optimization"]
  node_synthesizer["Synthetic data<br/>[synthesizer.py]"]
  node_simulator["Conversation simulation"]
  node_optimizer["Prompt optimization"]
end

subgraph group_platform["Configuration and integrations"]
  node_integrations["Framework integrations"]
  node_config["Settings<br/>[settings.py]"]
  node_cli["CLI<br/>[main.py]"]
  node_confident["Confident AI<br/>[api.py]"]
end

node_user(("Developer"))

node_user -->|"defines tests"| node_cases
node_user -->|"prepares data"| node_datasets
node_cases -->|"evaluates"| node_evaluate
node_datasets -->|"evaluates"| node_evaluate
node_evaluate -->|"scores with"| node_metrics
node_metrics -->|"judges with"| node_models
node_user -->|"instruments apps"| node_trace
node_integrations -->|"captures traces"| node_trace
node_user -->|"inspects traces"| node_inspect
node_user -->|"generates data"| node_synthesizer
node_user -->|"simulates chats"| node_simulator
node_user -->|"optimizes prompts"| node_optimizer
node_user -->|"configures"| node_cli
node_cli -->|"sets options"| node_config
node_user -->|"evaluates safety"| node_classifiers
node_evaluate -.->|"shares results"| node_confident

click node_cases "https://github.com/confident-ai/deepeval/blob/main/deepeval/test_case/llm_test_case.py"
click node_datasets "https://github.com/confident-ai/deepeval/blob/main/deepeval/dataset/dataset.py"
click node_evaluate "https://github.com/confident-ai/deepeval/blob/main/deepeval/evaluate/evaluate.py"
click node_metrics "https://github.com/confident-ai/deepeval/tree/main/deepeval/metrics"
click node_models "https://github.com/confident-ai/deepeval/tree/main/deepeval/models"
click node_classifiers "https://github.com/confident-ai/deepeval/tree/main/deepeval/classifiers"
click node_trace "https://github.com/confident-ai/deepeval/blob/main/deepeval/tracing/tracing.py"
click node_integrations "https://github.com/confident-ai/deepeval/tree/main/deepeval/integrations"
click node_inspect "https://github.com/confident-ai/deepeval/blob/main/deepeval/inspect/loader.py"
click node_synthesizer "https://github.com/confident-ai/deepeval/blob/main/deepeval/synthesizer/synthesizer.py"
click node_simulator "https://github.com/confident-ai/deepeval/blob/main/deepeval/simulator/conversation_simulator.py"
click node_optimizer "https://github.com/confident-ai/deepeval/blob/main/deepeval/optimizer/prompt_optimizer.py"
click node_config "https://github.com/confident-ai/deepeval/blob/main/deepeval/config/settings.py"
click node_cli "https://github.com/confident-ai/deepeval/blob/main/deepeval/cli/main.py"
click node_confident "https://github.com/confident-ai/deepeval/blob/main/deepeval/confident/api.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cases,node_datasets,node_evaluate toneBlue
class node_metrics,node_models,node_classifiers toneAmber
class node_trace,node_inspect toneMint
class node_synthesizer,node_simulator,node_optimizer toneRose
class node_integrations,node_config,node_cli,node_confident,node_user toneIndigo
```

## Components

### Evaluation workflow

- [Test cases](https://github.com/confident-ai/deepeval/blob/main/deepeval/test_case/llm_test_case.py): `deepeval/test_case/llm_test_case.py`
- [Datasets](https://github.com/confident-ai/deepeval/blob/main/deepeval/dataset/dataset.py): `deepeval/dataset/dataset.py`
- [Evaluation runner](https://github.com/confident-ai/deepeval/blob/main/deepeval/evaluate/evaluate.py): `deepeval/evaluate/evaluate.py`

### Judging and models

- [Evaluation metrics](https://github.com/confident-ai/deepeval/tree/main/deepeval/metrics): `deepeval/metrics`
- [Judge models](https://github.com/confident-ai/deepeval/tree/main/deepeval/models): `deepeval/models`
- [Safety classifiers](https://github.com/confident-ai/deepeval/tree/main/deepeval/classifiers): `deepeval/classifiers`

### Tracing and inspection

- [Tracing](https://github.com/confident-ai/deepeval/blob/main/deepeval/tracing/tracing.py): `deepeval/tracing/tracing.py`
- [Trace inspector](https://github.com/confident-ai/deepeval/blob/main/deepeval/inspect/loader.py): `deepeval/inspect/loader.py`

### Data and optimization

- [Synthetic data](https://github.com/confident-ai/deepeval/blob/main/deepeval/synthesizer/synthesizer.py): `deepeval/synthesizer/synthesizer.py`
- [Conversation simulation](https://github.com/confident-ai/deepeval/blob/main/deepeval/simulator/conversation_simulator.py): `deepeval/simulator/conversation_simulator.py`
- [Prompt optimization](https://github.com/confident-ai/deepeval/blob/main/deepeval/optimizer/prompt_optimizer.py): `deepeval/optimizer/prompt_optimizer.py`

### Configuration and integrations

- [Framework integrations](https://github.com/confident-ai/deepeval/tree/main/deepeval/integrations): `deepeval/integrations`
- [Settings](https://github.com/confident-ai/deepeval/blob/main/deepeval/config/settings.py): `deepeval/config/settings.py`
- [CLI](https://github.com/confident-ai/deepeval/blob/main/deepeval/cli/main.py): `deepeval/cli/main.py`
- [Confident AI](https://github.com/confident-ai/deepeval/blob/main/deepeval/confident/api.py): `deepeval/confident/api.py`

### Other components

- **Developer**

## Connections

- Developer → Test cases: defines tests (evidence: `README.md`)
- Developer → Datasets: prepares data (evidence: `README.md`)
- Test cases → Evaluation runner: evaluates (evidence: `README.md`)
- Datasets → Evaluation runner: evaluates (evidence: `README.md`)
- Evaluation runner → Evaluation metrics: scores with (evidence: `README.md`)
- Evaluation metrics → Judge models: judges with (evidence: `README.md`)
- Developer → Tracing: instruments apps (evidence: `README.md`)
- Framework integrations → Tracing: captures traces (evidence: `README.md`)
- Developer → Trace inspector: inspects traces (evidence: `README.md`)
- Developer → Synthetic data: generates data (evidence: `README.md`)
- Developer → Conversation simulation: simulates chats (evidence: `README.md`)
- Developer → Prompt optimization: optimizes prompts (evidence: `README.md`)
- Developer → CLI: configures (evidence: `README.md`)
- CLI → Settings: sets options (evidence: `README.md`)
- Developer → Safety classifiers: evaluates safety (evidence: `README.md`)
- Evaluation runner → Confident AI: shares results
