How awslabs/uaef works
UAEF is a framework-agnostic agent-evaluation product: agent traces and ground truth are normalized, evaluated across metric dimensions, and returned or optionally persisted; analysis and reporting support follow-up and comparisons. Its service adds a browser UI and asynchronous AWS evaluation jobs, with separate job-state storage and library result persistence. The README documents the library flow; source-index references and excerpts support the service links. Many library and service modules are unsampled, so uncertain internal call paths are omitted.
Main entry points: evaluation_result.py Evaluation Results · evaluation Evaluation Modes · metrics Metric Dimensions · api.py Service API
Components 21
Trace Inputs
- Framework Adapters
src/uaef/adapters - Canonical Traces
src/uaef/models/agent_trace.py - Ground Truth
src/uaef/models/ground_truth.py - Evaluation Input
src/uaef/models/evaluation_input.py
Evaluation Engine
- Evaluation API
src/uaef/api/__init__.py - Evaluation Modes
src/uaef/evaluation - Metric Dimensions
src/uaef/metrics - Metric Integrations
src/uaef/integrations - LLM Judge
src/uaef/llm_judge - Evaluation Results
src/uaef/models/evaluation_result.py
Analysis and Reporting
- Experiments
src/uaef/experiments - Analysis
src/uaef/analysis - Reports and Dashboards
src/uaef/reporting - Human Feedback
src/uaef/hitl
Evaluation Service
- Service UI
uaef-service/ui/app/src/App.jsx - Service API
uaef-service/handlers/api.py - Job Orchestrator
uaef-service/handlers/orchestrator.py - Evaluation Worker
uaef-service/handlers/worker.py
AWS State and Persistence
- Job State
uaef-service/job_state.py - Experiment Persistence
src/uaef/storage/dynamodb_s3.py
Other components
- User / Platform
Connections 26
- User / Platform → Framework Adapters: provides traces
README.md - Framework Adapters → Canonical Traces: normalizes to
README.md - User / Platform → Ground Truth: provides truth
README.md - Canonical Traces → Evaluation Input: forms input
src/uaef/models/evaluation_input.py - Ground Truth → Evaluation Input: forms input
src/uaef/models/evaluation_input.py - Evaluation Input → Evaluation API: submitted to
README.md - Evaluation API → Evaluation Modes: dispatches
src/uaef/api/__init__.py - Evaluation Modes → Metric Dimensions: scores with
README.md - Metric Dimensions → Metric Integrations: uses optionally
src/uaef/metrics/ragas_metrics.py - Metric Dimensions → LLM Judge: uses for judging
README.md - Evaluation Modes → Evaluation Results: produces
README.md - Evaluation Results → User / Platform: returns scores
README.md - Evaluation Results → Experiment Persistence: persists optionally
README.md - Evaluation Results → Analysis: analyzed by
README.md - Evaluation Results → Reports and Dashboards: reported by
README.md - Evaluation Results → Experiments: tracked in
README.md - Experiments → Reports and Dashboards: compared in
README.md - Evaluation Results → Human Feedback: reviewed through
README.md - User / Platform → Service UI: uses
uaef-service/ui/app/src/App.jsx - Service UI → Service API: requests evaluation
- Service API → Job State: manages jobs
uaef-service/handlers/api.py - Service API → Job Orchestrator: dispatches work
- Job Orchestrator → Job State: updates lifecycle
uaef-service/handlers/orchestrator.py - Job Orchestrator → Evaluation Worker: invokes
- Evaluation Worker → Evaluation API: runs evaluation
uaef-service/handlers/worker.py - Evaluation Worker → Experiment Persistence: persists results
uaef-service/handlers/worker.py