# judgmentlabs/judgeval architecture

> Architecture diagram of the GitHub repository judgmentlabs/judgeval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/judgmentlabs/judgeval
- Repository: https://github.com/judgmentlabs/judgeval

## Overview

Judgeval is a Python SDK for tracing LLM-agent behavior and improving agents through evaluation and monitoring. The primary workflow instruments an application, captures trace data, and sends it to Judgment; users can query that history with JQL, define prompt-based judges, and replay datasets through evaluations to validate changes. SDK client factories expose datasets, prompts, judges, and evaluation, while OpenTelemetry tracing and provider/framework wrappers capture runtime activity. The README documents online monitoring and alerts, but the sampled sources do not establish their implementation path; those edges are therefore omitted.

Main entry points: [`src/judgeval/internal/api/api_client.py`](https://github.com/judgmentlabs/judgeval/blob/main/src/judgeval/internal/api/api_client.py) (API client), [`src/judgeval/judgeval.py`](https://github.com/judgmentlabs/judgeval/blob/main/src/judgeval/judgeval.py) (Judgeval client), [`src/judgeval/offline_tests/offline_test_runner.py`](https://github.com/judgmentlabs/judgeval/blob/main/src/judgeval/offline_tests/offline_test_runner.py) (Offline replay), [`src/judgeval/evaluation`](https://github.com/judgmentlabs/judgeval/tree/main/src/judgeval/evaluation) (Evaluation runner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_capture["Trace Capture"]
  node_tracer["Tracer API<br/>[base_tracer.py]"]
  node_trace_pipeline["Trace processing"]
  node_trace_export["Span exporter"]
  node_llm_wrappers["LLM wrappers"]
  node_framework_wrappers["Framework adapters"]
end

subgraph group_improve["Evaluation &amp; Improvement"]
  node_dataset_factory["Dataset management<br/>[dataset_factory.py]"]
  node_offline_runner["Offline replay"]
  node_evaluation["Evaluation runner"]
  node_agent_judges["Agent judges"]
  node_prompts["Prompt management<br/>[prompt_factory.py]"]
  node_scorer_upload["Custom scorer upload<br/>[upload_judge.py]"]
end

subgraph group_client["SDK &amp; Queries"]
  node_client["Judgeval client<br/>[judgeval.py]"]
  node_jql["JQL builders<br/>[__init__.py]"]
end

subgraph group_transport["Platform Transport"]
  node_api["API client<br/>[api_client.py]"]
  node_trace_data["Trace and score data"]
end

node_app(("Agent application"))
node_providers(("LLM providers"))
node_judgment(("Judgment platform"))

node_app -->|"instruments"| node_tracer
node_tracer -->|"creates spans"| node_trace_pipeline
node_trace_pipeline -->|"exports spans"| node_trace_export
node_trace_export -->|"sends traces"| node_api
node_app -->|"wraps calls"| node_llm_wrappers
node_llm_wrappers -.->|"instruments"| node_providers
node_app -.->|"integrates"| node_framework_wrappers
node_framework_wrappers -->|"records activity"| node_tracer
node_client -->|"creates factory"| node_dataset_factory
node_client -->|"creates runner"| node_evaluation
node_client -->|"creates factory"| node_agent_judges
node_client -->|"creates factory"| node_prompts
node_client -->|"submits queries"| node_jql
node_jql -->|"query request"| node_api
node_dataset_factory -->|"manages datasets"| node_api
node_offline_runner -->|"fetches examples"| node_api
node_offline_runner -->|"replays examples"| node_app
node_offline_runner -->|"logs results"| node_api
node_evaluation -->|"runs evaluations"| node_api
node_evaluation -->|"builds results"| node_trace_data
node_agent_judges -->|"manages judges"| node_api
node_prompts -->|"manages prompts"| node_api
node_scorer_upload -->|"uploads bundle"| node_api
node_api -->|"HTTP requests"| node_judgment
node_api -->|"decodes responses"| node_trace_data

click node_tracer "https://github.com/judgmentlabs/judgeval/blob/main/src/judgeval/trace/base_tracer.py"
click node_trace_pipeline "https://github.com/judgmentlabs/judgeval/tree/main/src/judgeval/trace"
click node_trace_export "https://github.com/judgmentlabs/judgeval/blob/main/src/judgeval/trace/exporters/judgment_span_exporter.py"
click node_llm_wrappers "https://github.com/judgmentlabs/judgeval/tree/main/src/judgeval/instrumentation/llm"
click node_framework_wrappers "https://github.com/judgmentlabs/judgeval/tree/main/src/judgeval/integrations"
click node_dataset_factory "https://github.com/judgmentlabs/judgeval/blob/main/src/judgeval/datasets/dataset_factory.py"
click node_offline_runner "https://github.com/judgmentlabs/judgeval/blob/main/src/judgeval/offline_tests/offline_test_runner.py"
click node_evaluation "https://github.com/judgmentlabs/judgeval/tree/main/src/judgeval/evaluation"
click node_agent_judges "https://github.com/judgmentlabs/judgeval/blob/main/src/judgeval/agent_judges/agent_judge_factory.py"
click node_prompts "https://github.com/judgmentlabs/judgeval/blob/main/src/judgeval/prompts/prompt_factory.py"
click node_scorer_upload "https://github.com/judgmentlabs/judgeval/blob/main/src/judgeval/cli/upload_judge.py"
click node_client "https://github.com/judgmentlabs/judgeval/blob/main/src/judgeval/judgeval.py"
click node_jql "https://github.com/judgmentlabs/judgeval/blob/main/src/judgeval/jql/__init__.py"
click node_api "https://github.com/judgmentlabs/judgeval/blob/main/src/judgeval/internal/api/api_client.py"
click node_trace_data "https://github.com/judgmentlabs/judgeval/tree/main/src/judgeval/data"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_tracer,node_trace_pipeline,node_trace_export,node_llm_wrappers,node_framework_wrappers toneBlue
class node_dataset_factory,node_offline_runner,node_evaluation,node_agent_judges,node_prompts,node_scorer_upload toneAmber
class node_client,node_jql toneMint
class node_api,node_trace_data toneRose
class node_app,node_providers,node_judgment toneIndigo
```

## Components

### Trace Capture

- [Tracer API](https://github.com/judgmentlabs/judgeval/blob/main/src/judgeval/trace/base_tracer.py): `src/judgeval/trace/base_tracer.py`
- [Trace processing](https://github.com/judgmentlabs/judgeval/tree/main/src/judgeval/trace): `src/judgeval/trace`
- [Span exporter](https://github.com/judgmentlabs/judgeval/blob/main/src/judgeval/trace/exporters/judgment_span_exporter.py): `src/judgeval/trace/exporters/judgment_span_exporter.py`
- [LLM wrappers](https://github.com/judgmentlabs/judgeval/tree/main/src/judgeval/instrumentation/llm): `src/judgeval/instrumentation/llm`
- [Framework adapters](https://github.com/judgmentlabs/judgeval/tree/main/src/judgeval/integrations): `src/judgeval/integrations`

### Evaluation & Improvement

- [Dataset management](https://github.com/judgmentlabs/judgeval/blob/main/src/judgeval/datasets/dataset_factory.py): `src/judgeval/datasets/dataset_factory.py`
- [Offline replay](https://github.com/judgmentlabs/judgeval/blob/main/src/judgeval/offline_tests/offline_test_runner.py): `src/judgeval/offline_tests/offline_test_runner.py`
- [Evaluation runner](https://github.com/judgmentlabs/judgeval/tree/main/src/judgeval/evaluation): `src/judgeval/evaluation`
- [Agent judges](https://github.com/judgmentlabs/judgeval/blob/main/src/judgeval/agent_judges/agent_judge_factory.py): `src/judgeval/agent_judges/agent_judge_factory.py`
- [Prompt management](https://github.com/judgmentlabs/judgeval/blob/main/src/judgeval/prompts/prompt_factory.py): `src/judgeval/prompts/prompt_factory.py`
- [Custom scorer upload](https://github.com/judgmentlabs/judgeval/blob/main/src/judgeval/cli/upload_judge.py): `src/judgeval/cli/upload_judge.py`

### SDK & Queries

- [Judgeval client](https://github.com/judgmentlabs/judgeval/blob/main/src/judgeval/judgeval.py): `src/judgeval/judgeval.py`
- [JQL builders](https://github.com/judgmentlabs/judgeval/blob/main/src/judgeval/jql/__init__.py): `src/judgeval/jql/__init__.py`

### Platform Transport

- [API client](https://github.com/judgmentlabs/judgeval/blob/main/src/judgeval/internal/api/api_client.py): `src/judgeval/internal/api/api_client.py`
- [Trace and score data](https://github.com/judgmentlabs/judgeval/tree/main/src/judgeval/data): `src/judgeval/data`

### Other components

- **Agent application**
- **LLM providers**
- **Judgment platform**

## Connections

- Agent application → Tracer API: instruments
- Tracer API → Trace processing: creates spans
- Trace processing → Span exporter: exports spans
- Span exporter → API client: sends traces
- Agent application → LLM wrappers: wraps calls
- LLM wrappers → LLM providers: instruments
- Agent application → Framework adapters: integrates
- Framework adapters → Tracer API: records activity
- Judgeval client → Dataset management: creates factory
- Judgeval client → Evaluation runner: creates runner
- Judgeval client → Agent judges: creates factory
- Judgeval client → Prompt management: creates factory
- Judgeval client → JQL builders: submits queries
- JQL builders → API client: query request
- Dataset management → API client: manages datasets
- Offline replay → API client: fetches examples
- Offline replay → Agent application: replays examples
- Offline replay → API client: logs results
- Evaluation runner → API client: runs evaluations
- Evaluation runner → Trace and score data: builds results
- Agent judges → API client: manages judges
- Prompt management → API client: manages prompts
- Custom scorer upload → API client: uploads bundle
- API client → Judgment platform: HTTP requests
- API client → Trace and score data: decodes responses