# aws-samples/sample-gen-ai-evaluations-workshop architecture

> Architecture diagram of the GitHub repository aws-samples/sample-gen-ai-evaluations-workshop, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aws-samples/sample-gen-ai-evaluations-workshop
- Repository: https://github.com/aws-samples/sample-gen-ai-evaluations-workshop

## Overview

This repository is a hands-on Generative AI evaluation workshop, not one unified runtime application. Its main learning path progresses through foundational evaluation topics, then branches into workload-, framework-, and industry-specific modules; interactive learning offers an alternative guided route. Representative internal implementations include quality scoring, model configuration, AgentCore examples, and a speech-to-speech demo/evaluation stack. The README documents many modules whose source was not sampled, so those are represented at their existing directory paths without inferred internal call chains.

Main entry points: [`Workload Specific Evaluations`](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Workload%20Specific%20Evaluations) (Workload modules), [`Framework Specific Evaluations`](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Framework%20Specific%20Evaluations) (Framework modules), [`Workload Specific Evaluations/Speech to Speech/sample_s2s_app/python-server/server.py`](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/blob/main/Workload%20Specific%20Evaluations/Speech%20to%20Speech/sample_s2s_app/python-server/server.py) (WebSocket server), [`Foundational Evaluations`](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Foundational%20Evaluations) (Foundational modules)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_learning["Workshop learning paths"]
  node_foundations["Foundational modules"]
  node_quality["Quality metrics<br/>[utils.py]"]
  node_model_config["Model configuration<br/>[model_config.py]"]
  node_interactive["Interactive learning"]
end

subgraph group_workloads["Workload evaluations"]
  node_workload_catalog["Workload modules"]
  node_rag["RAG evaluation"]
  node_guardrails["Guardrails and red team"]
  node_tool_calling["Tool and MCP calling"]
  node_capacity["Capacity management"]
  node_multiagent["Shared-context metrics"]
end

subgraph group_frameworks["Framework evaluations"]
  node_framework_catalog["Framework modules"]
  node_agentcore["AgentCore evaluation"]
  node_dspy["DSPy optimization"]
  node_mlflow["MLflow tracking"]
  node_deepeval["DeepEval metrics"]
end

subgraph group_industry["Industry evaluations"]
  node_retail["Retail and CPG"]
end

subgraph group_speech["Speech-to-speech demo"]
  node_speech_ui["Conversation playground<br/>[s2s.js]"]
  node_speech_server["WebSocket server<br/>[server.py]"]
  node_session_manager["Stream sessions"]
  node_transitions["Session transitions"]
  node_speech_eval["Speech evaluator<br/>[s2s_evaluator.py]"]
  node_speech_annotation["Annotation interface"]
end

node_learner(("Learner"))

node_learner -->|"works through"| node_foundations
node_learner -->|"can choose"| node_interactive
node_foundations -->|"includes"| node_quality
node_quality -->|"imports"| node_model_config
node_learner -->|"then selects"| node_workload_catalog
node_workload_catalog -->|"offers"| node_rag
node_workload_catalog -->|"offers"| node_guardrails
node_workload_catalog -->|"offers"| node_tool_calling
node_workload_catalog -->|"offers"| node_capacity
node_workload_catalog -->|"offers"| node_multiagent
node_learner -->|"can select"| node_framework_catalog
node_framework_catalog -->|"offers"| node_agentcore
node_framework_catalog -->|"offers"| node_dspy
node_framework_catalog -->|"offers"| node_mlflow
node_framework_catalog -->|"offers"| node_deepeval
node_learner -->|"can select"| node_retail
node_speech_ui -.->|"sends events"| node_speech_server
node_speech_server -->|"imports and manages"| node_session_manager
node_speech_server -->|"imports and manages"| node_transitions
node_session_manager -.->|"used by orchestration"| node_transitions
node_speech_eval -.->|"supports evaluation workflow"| node_speech_annotation
node_workload_catalog -->|"includes speech evaluation"| node_speech_eval

click node_foundations "https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Foundational%20Evaluations"
click node_quality "https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/blob/main/Foundational%20Evaluations/02-quality-metrics/utils.py"
click node_model_config "https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/blob/main/Foundational%20Evaluations/model_config.py"
click node_interactive "https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Interactive%20Learning"
click node_workload_catalog "https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Workload%20Specific%20Evaluations"
click node_rag "https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Workload%20Specific%20Evaluations/Basic%20RAG"
click node_guardrails "https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Workload%20Specific%20Evaluations/Guardrails"
click node_tool_calling "https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Workload%20Specific%20Evaluations/MCP%20Tool%20Calling"
click node_capacity "https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Workload%20Specific%20Evaluations/Capacity%20Management"
click node_multiagent "https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/blob/main/Workload%20Specific%20Evaluations/Multiagent%20Shared%20Context%20Evaluation/metrics_collector.py"
click node_framework_catalog "https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Framework%20Specific%20Evaluations"
click node_agentcore "https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Framework%20Specific%20Evaluations/AgentCore"
click node_dspy "https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Framework%20Specific%20Evaluations/DSPy"
click node_mlflow "https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Framework%20Specific%20Evaluations/MLflow"
click node_deepeval "https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Framework%20Specific%20Evaluations/DeepEval"
click node_retail "https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Industry%20Specific%20Evaluations/Retail%20and%20CPG"
click node_speech_ui "https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/blob/main/Workload%20Specific%20Evaluations/Speech%20to%20Speech/sample_s2s_app/react-client/src/s2s.js"
click node_speech_server "https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/blob/main/Workload%20Specific%20Evaluations/Speech%20to%20Speech/sample_s2s_app/python-server/server.py"
click node_session_manager "https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/blob/main/Workload%20Specific%20Evaluations/Speech%20to%20Speech/sample_s2s_app/python-server/s2s_session_manager.py"
click node_transitions "https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/blob/main/Workload%20Specific%20Evaluations/Speech%20to%20Speech/sample_s2s_app/python-server/session_transition_manager.py"
click node_speech_eval "https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/blob/main/Workload%20Specific%20Evaluations/Speech%20to%20Speech/s2s_evaluator.py"
click node_speech_annotation "https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/blob/main/Workload%20Specific%20Evaluations/Speech%20to%20Speech/s2s_annotation_ui.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_foundations,node_quality,node_model_config,node_interactive toneBlue
class node_workload_catalog,node_rag,node_guardrails,node_tool_calling,node_capacity,node_multiagent toneAmber
class node_framework_catalog,node_agentcore,node_dspy,node_mlflow,node_deepeval toneMint
class node_retail toneRose
class node_speech_ui,node_speech_server,node_session_manager,node_transitions,node_speech_eval,node_speech_annotation,node_learner toneIndigo
```

## Components

### Workshop learning paths

- [Foundational modules](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Foundational%20Evaluations): `Foundational Evaluations`
- [Quality metrics](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/blob/main/Foundational%20Evaluations/02-quality-metrics/utils.py): `Foundational Evaluations/02-quality-metrics/utils.py`
- [Model configuration](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/blob/main/Foundational%20Evaluations/model_config.py): `Foundational Evaluations/model_config.py`
- [Interactive learning](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Interactive%20Learning): `Interactive Learning`

### Workload evaluations

- [Workload modules](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Workload%20Specific%20Evaluations): `Workload Specific Evaluations`
- [RAG evaluation](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Workload%20Specific%20Evaluations/Basic%20RAG): `Workload Specific Evaluations/Basic RAG`
- [Guardrails and red team](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Workload%20Specific%20Evaluations/Guardrails): `Workload Specific Evaluations/Guardrails`
- [Tool and MCP calling](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Workload%20Specific%20Evaluations/MCP%20Tool%20Calling): `Workload Specific Evaluations/MCP Tool Calling`
- [Capacity management](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Workload%20Specific%20Evaluations/Capacity%20Management): `Workload Specific Evaluations/Capacity Management`
- [Shared-context metrics](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/blob/main/Workload%20Specific%20Evaluations/Multiagent%20Shared%20Context%20Evaluation/metrics_collector.py): `Workload Specific Evaluations/Multiagent Shared Context Evaluation/metrics_collector.py`

### Framework evaluations

- [Framework modules](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Framework%20Specific%20Evaluations): `Framework Specific Evaluations`
- [AgentCore evaluation](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Framework%20Specific%20Evaluations/AgentCore): `Framework Specific Evaluations/AgentCore`
- [DSPy optimization](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Framework%20Specific%20Evaluations/DSPy): `Framework Specific Evaluations/DSPy`
- [MLflow tracking](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Framework%20Specific%20Evaluations/MLflow): `Framework Specific Evaluations/MLflow`
- [DeepEval metrics](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Framework%20Specific%20Evaluations/DeepEval): `Framework Specific Evaluations/DeepEval`

### Industry evaluations

- [Retail and CPG](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/tree/main/Industry%20Specific%20Evaluations/Retail%20and%20CPG): `Industry Specific Evaluations/Retail and CPG`

### Speech-to-speech demo

- [Conversation playground](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/blob/main/Workload%20Specific%20Evaluations/Speech%20to%20Speech/sample_s2s_app/react-client/src/s2s.js): `Workload Specific Evaluations/Speech to Speech/sample_s2s_app/react-client/src/s2s.js`
- [WebSocket server](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/blob/main/Workload%20Specific%20Evaluations/Speech%20to%20Speech/sample_s2s_app/python-server/server.py): `Workload Specific Evaluations/Speech to Speech/sample_s2s_app/python-server/server.py`
- [Stream sessions](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/blob/main/Workload%20Specific%20Evaluations/Speech%20to%20Speech/sample_s2s_app/python-server/s2s_session_manager.py): `Workload Specific Evaluations/Speech to Speech/sample_s2s_app/python-server/s2s_session_manager.py`
- [Session transitions](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/blob/main/Workload%20Specific%20Evaluations/Speech%20to%20Speech/sample_s2s_app/python-server/session_transition_manager.py): `Workload Specific Evaluations/Speech to Speech/sample_s2s_app/python-server/session_transition_manager.py`
- [Speech evaluator](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/blob/main/Workload%20Specific%20Evaluations/Speech%20to%20Speech/s2s_evaluator.py): `Workload Specific Evaluations/Speech to Speech/s2s_evaluator.py`
- [Annotation interface](https://github.com/aws-samples/sample-gen-ai-evaluations-workshop/blob/main/Workload%20Specific%20Evaluations/Speech%20to%20Speech/s2s_annotation_ui.py): `Workload Specific Evaluations/Speech to Speech/s2s_annotation_ui.py`

### Other components

- **Learner**

## Connections

- Learner → Foundational modules: works through (evidence: `README.md`)
- Learner → Interactive learning: can choose (evidence: `README.md`)
- Foundational modules → Quality metrics: includes (evidence: `README.md`)
- Quality metrics → Model configuration: imports (evidence: `Foundational Evaluations/02-quality-metrics/utils.py`)
- Learner → Workload modules: then selects (evidence: `README.md`)
- Workload modules → RAG evaluation: offers (evidence: `README.md`)
- Workload modules → Guardrails and red team: offers (evidence: `README.md`)
- Workload modules → Tool and MCP calling: offers (evidence: `README.md`)
- Workload modules → Capacity management: offers (evidence: `README.md`)
- Workload modules → Shared-context metrics: offers (evidence: `README.md`)
- Learner → Framework modules: can select (evidence: `README.md`)
- Framework modules → AgentCore evaluation: offers (evidence: `README.md`)
- Framework modules → DSPy optimization: offers (evidence: `README.md`)
- Framework modules → MLflow tracking: offers (evidence: `README.md`)
- Framework modules → DeepEval metrics: offers (evidence: `README.md`)
- Learner → Retail and CPG: can select (evidence: `README.md`)
- Conversation playground → WebSocket server: sends events
- WebSocket server → Stream sessions: imports and manages (evidence: `Workload Specific Evaluations/Speech to Speech/sample_s2s_app/python-server/server.py`)
- WebSocket server → Session transitions: imports and manages (evidence: `Workload Specific Evaluations/Speech to Speech/sample_s2s_app/python-server/server.py`)
- Stream sessions → Session transitions: used by orchestration
- Speech evaluator → Annotation interface: supports evaluation workflow
- Workload modules → Speech evaluator: includes speech evaluation (evidence: `README.md`)