How aws-samples/sample-gen-ai-evaluations-workshop works
This repository is a hands-on Generative AI evaluation workshop, not one unified runtime application. Its main learning path progresses through foundational evaluation topics, then branches into workload-, framework-, and industry-specific modules; interactive learning offers an alternative guided route. Representative internal implementations include quality scoring, model configuration, AgentCore examples, and a speech-to-speech demo/evaluation stack. The README documents many modules whose source was not sampled, so those are represented at their existing directory paths without inferred internal call chains.
Main entry points: Workload Specific Evaluations Workload modules · Framework Specific Evaluations Framework modules · server.py WebSocket server · Foundational Evaluations Foundational modules
Components 23
Workshop learning paths
- Foundational modules
Foundational Evaluations - Quality metrics
Foundational Evaluations/02-quality-metrics/utils.py - Model configuration
Foundational Evaluations/model_config.py - Interactive learning
Interactive Learning
Workload evaluations
- Workload modules
Workload Specific Evaluations - RAG evaluation
Workload Specific Evaluations/Basic RAG - Guardrails and red team
Workload Specific Evaluations/Guardrails - Tool and MCP calling
Workload Specific Evaluations/MCP Tool Calling - Capacity management
Workload Specific Evaluations/Capacity Management - Shared-context metrics
Workload Specific Evaluations/Multiagent Shared Context Evaluation/metrics_collector.py
Framework evaluations
- Framework modules
Framework Specific Evaluations - AgentCore evaluation
Framework Specific Evaluations/AgentCore - DSPy optimization
Framework Specific Evaluations/DSPy - MLflow tracking
Framework Specific Evaluations/MLflow - DeepEval metrics
Framework Specific Evaluations/DeepEval
Industry evaluations
- Retail and CPG
Industry Specific Evaluations/Retail and CPG
Speech-to-speech demo
- Conversation playground
Workload Specific Evaluations/Speech to Speech/sample_s2s_app/react-client/src/s2s.js - WebSocket server
Workload Specific Evaluations/Speech to Speech/sample_s2s_app/python-server/server.py - Stream sessions
Workload Specific Evaluations/Speech to Speech/sample_s2s_app/python-server/s2s_session_manager.py - Session transitions
Workload Specific Evaluations/Speech to Speech/sample_s2s_app/python-server/session_transition_manager.py - Speech evaluator
Workload Specific Evaluations/Speech to Speech/s2s_evaluator.py - Annotation interface
Workload Specific Evaluations/Speech to Speech/s2s_annotation_ui.py
Other components
- Learner
Connections 22
- Learner → Foundational modules: works through
README.md - Learner → Interactive learning: can choose
README.md - Foundational modules → Quality metrics: includes
README.md - Quality metrics → Model configuration: imports
Foundational Evaluations/02-quality-metrics/utils.py - Learner → Workload modules: then selects
README.md - Workload modules → RAG evaluation: offers
README.md - Workload modules → Guardrails and red team: offers
README.md - Workload modules → Tool and MCP calling: offers
README.md - Workload modules → Capacity management: offers
README.md - Workload modules → Shared-context metrics: offers
README.md - Learner → Framework modules: can select
README.md - Framework modules → AgentCore evaluation: offers
README.md - Framework modules → DSPy optimization: offers
README.md - Framework modules → MLflow tracking: offers
README.md - Framework modules → DeepEval metrics: offers
README.md - Learner → Retail and CPG: can select
README.md - Conversation playground → WebSocket server: sends events
- WebSocket server → Stream sessions: imports and manages
Workload Specific Evaluations/Speech to Speech/sample_s2s_app/python-server/server.py - WebSocket server → Session transitions: imports and manages
Workload Specific Evaluations/Speech to Speech/sample_s2s_app/python-server/server.py - Stream sessions → Session transitions: used by orchestration
- Speech evaluator → Annotation interface: supports evaluation workflow
- Workload modules → Speech evaluator: includes speech evaluation
README.md