# awslabs/uaef architecture

> Architecture diagram of the GitHub repository awslabs/uaef, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/awslabs/uaef
- Repository: https://github.com/awslabs/uaef

## Overview

UAEF is a framework-agnostic agent-evaluation product: agent traces and ground truth are normalized, evaluated across metric dimensions, and returned or optionally persisted; analysis and reporting support follow-up and comparisons. Its service adds a browser UI and asynchronous AWS evaluation jobs, with separate job-state storage and library result persistence. The README documents the library flow; source-index references and excerpts support the service links. Many library and service modules are unsampled, so uncertain internal call paths are omitted.

Main entry points: [`src/uaef/models/evaluation_result.py`](https://github.com/awslabs/uaef/blob/main/src/uaef/models/evaluation_result.py) (Evaluation Results), [`src/uaef/evaluation`](https://github.com/awslabs/uaef/tree/main/src/uaef/evaluation) (Evaluation Modes), [`src/uaef/metrics`](https://github.com/awslabs/uaef/tree/main/src/uaef/metrics) (Metric Dimensions), [`uaef-service/handlers/api.py`](https://github.com/awslabs/uaef/blob/main/uaef-service/handlers/api.py) (Service API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inputs["Trace Inputs"]
  node_adapters["Framework Adapters"]
  node_trace["Canonical Traces<br/>[agent_trace.py]"]
  node_groundtruth["Ground Truth<br/>[ground_truth.py]"]
  node_evaluation_input["Evaluation Input"]
end

subgraph group_evaluation["Evaluation Engine"]
  node_api["Evaluation API<br/>[__init__.py]"]
  node_evaluators["Evaluation Modes"]
  node_metrics["Metric Dimensions"]
  node_integrations["Metric Integrations"]
  node_judge["LLM Judge"]
  node_result["Evaluation Results"]
end

subgraph group_insight["Analysis and Reporting"]
  node_experiments["Experiments"]
  node_analysis["Analysis"]
  node_reporting["Reports and Dashboards"]
  node_feedback["Human Feedback"]
end

subgraph group_service["Evaluation Service"]
  node_ui["Service UI<br/>[App.jsx]"]
  node_service_api["Service API<br/>[api.py]"]
  node_orchestrator["Job Orchestrator<br/>[orchestrator.py]"]
  node_worker["Evaluation Worker<br/>[worker.py]"]
end

subgraph group_aws["AWS State and Persistence"]
  node_jobstate[("Job State<br/>[job_state.py]")]
  node_libstorage[("Experiment Persistence<br/>[dynamodb_s3.py]")]
end

node_user(("User / Platform"))

node_user -->|"provides traces"| node_adapters
node_adapters -->|"normalizes to"| node_trace
node_user -->|"provides truth"| node_groundtruth
node_trace -->|"forms input"| node_evaluation_input
node_groundtruth -->|"forms input"| node_evaluation_input
node_evaluation_input -->|"submitted to"| node_api
node_api -->|"dispatches"| node_evaluators
node_evaluators -->|"scores with"| node_metrics
node_metrics -.->|"uses optionally"| node_integrations
node_metrics -.->|"uses for judging"| node_judge
node_evaluators -->|"produces"| node_result
node_result -->|"returns scores"| node_user
node_result -.->|"persists optionally"| node_libstorage
node_result -->|"analyzed by"| node_analysis
node_result -->|"reported by"| node_reporting
node_result -->|"tracked in"| node_experiments
node_experiments -.->|"compared in"| node_reporting
node_result -.->|"reviewed through"| node_feedback
node_user -->|"uses"| node_ui
node_ui -.->|"requests evaluation"| node_service_api
node_service_api -->|"manages jobs"| node_jobstate
node_service_api -.->|"dispatches work"| node_orchestrator
node_orchestrator -->|"updates lifecycle"| node_jobstate
node_orchestrator -.->|"invokes"| node_worker
node_worker -->|"runs evaluation"| node_api
node_worker -->|"persists results"| node_libstorage

click node_adapters "https://github.com/awslabs/uaef/tree/main/src/uaef/adapters"
click node_trace "https://github.com/awslabs/uaef/blob/main/src/uaef/models/agent_trace.py"
click node_groundtruth "https://github.com/awslabs/uaef/blob/main/src/uaef/models/ground_truth.py"
click node_evaluation_input "https://github.com/awslabs/uaef/blob/main/src/uaef/models/evaluation_input.py"
click node_api "https://github.com/awslabs/uaef/blob/main/src/uaef/api/__init__.py"
click node_evaluators "https://github.com/awslabs/uaef/tree/main/src/uaef/evaluation"
click node_metrics "https://github.com/awslabs/uaef/tree/main/src/uaef/metrics"
click node_integrations "https://github.com/awslabs/uaef/tree/main/src/uaef/integrations"
click node_judge "https://github.com/awslabs/uaef/tree/main/src/uaef/llm_judge"
click node_result "https://github.com/awslabs/uaef/blob/main/src/uaef/models/evaluation_result.py"
click node_experiments "https://github.com/awslabs/uaef/tree/main/src/uaef/experiments"
click node_analysis "https://github.com/awslabs/uaef/tree/main/src/uaef/analysis"
click node_reporting "https://github.com/awslabs/uaef/tree/main/src/uaef/reporting"
click node_feedback "https://github.com/awslabs/uaef/tree/main/src/uaef/hitl"
click node_ui "https://github.com/awslabs/uaef/blob/main/uaef-service/ui/app/src/App.jsx"
click node_service_api "https://github.com/awslabs/uaef/blob/main/uaef-service/handlers/api.py"
click node_orchestrator "https://github.com/awslabs/uaef/blob/main/uaef-service/handlers/orchestrator.py"
click node_worker "https://github.com/awslabs/uaef/blob/main/uaef-service/handlers/worker.py"
click node_jobstate "https://github.com/awslabs/uaef/blob/main/uaef-service/job_state.py"
click node_libstorage "https://github.com/awslabs/uaef/blob/main/src/uaef/storage/dynamodb_s3.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_adapters,node_trace,node_groundtruth,node_evaluation_input,node_user toneBlue
class node_api,node_evaluators,node_metrics,node_integrations,node_judge,node_result toneAmber
class node_experiments,node_analysis,node_reporting,node_feedback toneMint
class node_ui,node_service_api,node_orchestrator,node_worker toneRose
class node_jobstate,node_libstorage toneIndigo
```

## Components

### Trace Inputs

- [Framework Adapters](https://github.com/awslabs/uaef/tree/main/src/uaef/adapters): `src/uaef/adapters`
- [Canonical Traces](https://github.com/awslabs/uaef/blob/main/src/uaef/models/agent_trace.py): `src/uaef/models/agent_trace.py`
- [Ground Truth](https://github.com/awslabs/uaef/blob/main/src/uaef/models/ground_truth.py): `src/uaef/models/ground_truth.py`
- [Evaluation Input](https://github.com/awslabs/uaef/blob/main/src/uaef/models/evaluation_input.py): `src/uaef/models/evaluation_input.py`

### Evaluation Engine

- [Evaluation API](https://github.com/awslabs/uaef/blob/main/src/uaef/api/__init__.py): `src/uaef/api/__init__.py`
- [Evaluation Modes](https://github.com/awslabs/uaef/tree/main/src/uaef/evaluation): `src/uaef/evaluation`
- [Metric Dimensions](https://github.com/awslabs/uaef/tree/main/src/uaef/metrics): `src/uaef/metrics`
- [Metric Integrations](https://github.com/awslabs/uaef/tree/main/src/uaef/integrations): `src/uaef/integrations`
- [LLM Judge](https://github.com/awslabs/uaef/tree/main/src/uaef/llm_judge): `src/uaef/llm_judge`
- [Evaluation Results](https://github.com/awslabs/uaef/blob/main/src/uaef/models/evaluation_result.py): `src/uaef/models/evaluation_result.py`

### Analysis and Reporting

- [Experiments](https://github.com/awslabs/uaef/tree/main/src/uaef/experiments): `src/uaef/experiments`
- [Analysis](https://github.com/awslabs/uaef/tree/main/src/uaef/analysis): `src/uaef/analysis`
- [Reports and Dashboards](https://github.com/awslabs/uaef/tree/main/src/uaef/reporting): `src/uaef/reporting`
- [Human Feedback](https://github.com/awslabs/uaef/tree/main/src/uaef/hitl): `src/uaef/hitl`

### Evaluation Service

- [Service UI](https://github.com/awslabs/uaef/blob/main/uaef-service/ui/app/src/App.jsx): `uaef-service/ui/app/src/App.jsx`
- [Service API](https://github.com/awslabs/uaef/blob/main/uaef-service/handlers/api.py): `uaef-service/handlers/api.py`
- [Job Orchestrator](https://github.com/awslabs/uaef/blob/main/uaef-service/handlers/orchestrator.py): `uaef-service/handlers/orchestrator.py`
- [Evaluation Worker](https://github.com/awslabs/uaef/blob/main/uaef-service/handlers/worker.py): `uaef-service/handlers/worker.py`

### AWS State and Persistence

- [Job State](https://github.com/awslabs/uaef/blob/main/uaef-service/job_state.py): `uaef-service/job_state.py`
- [Experiment Persistence](https://github.com/awslabs/uaef/blob/main/src/uaef/storage/dynamodb_s3.py): `src/uaef/storage/dynamodb_s3.py`

### Other components

- **User / Platform**

## Connections

- User / Platform → Framework Adapters: provides traces (evidence: `README.md`)
- Framework Adapters → Canonical Traces: normalizes to (evidence: `README.md`)
- User / Platform → Ground Truth: provides truth (evidence: `README.md`)
- Canonical Traces → Evaluation Input: forms input (evidence: `src/uaef/models/evaluation_input.py`)
- Ground Truth → Evaluation Input: forms input (evidence: `src/uaef/models/evaluation_input.py`)
- Evaluation Input → Evaluation API: submitted to (evidence: `README.md`)
- Evaluation API → Evaluation Modes: dispatches (evidence: `src/uaef/api/__init__.py`)
- Evaluation Modes → Metric Dimensions: scores with (evidence: `README.md`)
- Metric Dimensions → Metric Integrations: uses optionally (evidence: `src/uaef/metrics/ragas_metrics.py`)
- Metric Dimensions → LLM Judge: uses for judging (evidence: `README.md`)
- Evaluation Modes → Evaluation Results: produces (evidence: `README.md`)
- Evaluation Results → User / Platform: returns scores (evidence: `README.md`)
- Evaluation Results → Experiment Persistence: persists optionally (evidence: `README.md`)
- Evaluation Results → Analysis: analyzed by (evidence: `README.md`)
- Evaluation Results → Reports and Dashboards: reported by (evidence: `README.md`)
- Evaluation Results → Experiments: tracked in (evidence: `README.md`)
- Experiments → Reports and Dashboards: compared in (evidence: `README.md`)
- Evaluation Results → Human Feedback: reviewed through (evidence: `README.md`)
- User / Platform → Service UI: uses (evidence: `uaef-service/ui/app/src/App.jsx`)
- Service UI → Service API: requests evaluation
- Service API → Job State: manages jobs (evidence: `uaef-service/handlers/api.py`)
- Service API → Job Orchestrator: dispatches work
- Job Orchestrator → Job State: updates lifecycle (evidence: `uaef-service/handlers/orchestrator.py`)
- Job Orchestrator → Evaluation Worker: invokes
- Evaluation Worker → Evaluation API: runs evaluation (evidence: `uaef-service/handlers/worker.py`)
- Evaluation Worker → Experiment Persistence: persists results (evidence: `uaef-service/handlers/worker.py`)
