How judgmentlabs/judgeval works
Judgeval is a Python SDK for tracing LLM-agent behavior and improving agents through evaluation and monitoring. The primary workflow instruments an application, captures trace data, and sends it to Judgment; users can query that history with JQL, define prompt-based judges, and replay datasets through evaluations to validate changes. SDK client factories expose datasets, prompts, judges, and evaluation, while OpenTelemetry tracing and provider/framework wrappers capture runtime activity. The README documents online monitoring and alerts, but the sampled sources do not establish their implementation path; those edges are therefore omitted.
Main entry points: api_client.py API client · judgeval.py Judgeval client · offline_test_runner.py Offline replay · evaluation Evaluation runner
Components 18
Trace Capture
- Tracer API
src/judgeval/trace/base_tracer.py - Trace processing
src/judgeval/trace - Span exporter
src/judgeval/trace/exporters/judgment_span_exporter.py - LLM wrappers
src/judgeval/instrumentation/llm - Framework adapters
src/judgeval/integrations
Evaluation & Improvement
- Dataset management
src/judgeval/datasets/dataset_factory.py - Offline replay
src/judgeval/offline_tests/offline_test_runner.py - Evaluation runner
src/judgeval/evaluation - Agent judges
src/judgeval/agent_judges/agent_judge_factory.py - Prompt management
src/judgeval/prompts/prompt_factory.py - Custom scorer upload
src/judgeval/cli/upload_judge.py
SDK & Queries
- Judgeval client
src/judgeval/judgeval.py - JQL builders
src/judgeval/jql/__init__.py
Platform Transport
- API client
src/judgeval/internal/api/api_client.py - Trace and score data
src/judgeval/data
Other components
- Agent application
- LLM providers
- Judgment platform
Connections 25
- Agent application → Tracer API: instruments
- Tracer API → Trace processing: creates spans
- Trace processing → Span exporter: exports spans
- Span exporter → API client: sends traces
- Agent application → LLM wrappers: wraps calls
- LLM wrappers → LLM providers: instruments
- Agent application → Framework adapters: integrates
- Framework adapters → Tracer API: records activity
- Judgeval client → Dataset management: creates factory
- Judgeval client → Evaluation runner: creates runner
- Judgeval client → Agent judges: creates factory
- Judgeval client → Prompt management: creates factory
- Judgeval client → JQL builders: submits queries
- JQL builders → API client: query request
- Dataset management → API client: manages datasets
- Offline replay → API client: fetches examples
- Offline replay → Agent application: replays examples
- Offline replay → API client: logs results
- Evaluation runner → API client: runs evaluations
- Evaluation runner → Trace and score data: builds results
- Agent judges → API client: manages judges
- Prompt management → API client: manages prompts
- Custom scorer upload → API client: uploads bundle
- API client → Judgment platform: HTTP requests
- API client → Trace and score data: decodes responses