How ziadisafouene/multiturn-agent-evaluator works
multiturn-agent-eval is a framework for stress-testing multi-turn conversational agents. Users configure an evaluation through the CLI or Python API, supply scenarios or a natural-language specification, generate persona/language/mutation test cases, simulate conversations, and score results with deterministic checks and an optional LLM judge. The graph preserves the documented synthesis, multilingual, adversarial, and dual-scoring capabilities. Some orchestration and report-writing details are only partially excerpted, so uncertain internal links are omitted.
Main entry points: orchestrator.py Evaluation coordinator · simulator.py Conversation simulator · generator.py Scenario synthesis · tester.py Tester LLM client
Components 21
Entry and coordination
- CLI
core/cli.py - Evaluation coordinator
core/orchestrator.py
Scenario definition
- Scenario input
- Scenario synthesis
core/scenario/generator.py - Scenario parser
core/scenario/parser.py - Scenario validation
core/scenario/validator.py - Scenario JSON storage
core/scenario/storage.py - State machine
core/scenario/state_machine.py
Test case design
- Mutation generator
core/mutation/generator.py - Language resources
core/data/multilingual.py
Conversation runtime
- Conversation simulator
core/simulation/simulator.py - Agent adapter
core/simulation/agent.py - Persona rewriting
core/persona/engine.py - Prompt templates
core/llm/prompts.py - Agent LLM client
core/llm/client.py - Tester LLM client
core/llm/tester.py
Evaluation and models
- Rule and judge scoring
core/evaluation/engine.py - Domain and result models
core/utils/models.py
Other components
- Evaluator
- OpenRouter
- Evaluation results
Connections 27
- Evaluator → CLI: runs
README.md - Evaluator → Evaluation coordinator: configures
README.md - CLI → Evaluation coordinator: invokes
core/cli.py - Evaluation coordinator → Scenario input: accepts
README.md - Scenario input → Scenario JSON storage: loads JSON
core/scenario/storage.py - Scenario JSON storage → Scenario parser: parses
core/scenario/storage.py - Evaluation coordinator → Scenario synthesis: requests synthesis
core/orchestrator.py - Scenario synthesis → Tester LLM client: generates with
core/scenario/generator.py - Scenario synthesis → Scenario parser: parses output
core/scenario/generator.py - Scenario synthesis → Scenario validation: validates
core/scenario/generator.py - Evaluation coordinator → Mutation generator: generates cases
core/orchestrator.py - Evaluation coordinator → Conversation simulator: runs simulations
core/orchestrator.py - Conversation simulator → Agent adapter: uses
core/simulation/agent.py - Agent adapter → Agent LLM client: requests responses
core/simulation/agent.py - Conversation simulator → Agent LLM client: queries
core/simulation/simulator.py - Conversation simulator → Persona rewriting: rewrites user turns
core/simulation/simulator.py - Persona rewriting → Tester LLM client: generates rewrite
core/persona/engine.py - Persona rewriting → Prompt templates: builds prompt
core/persona/engine.py - Conversation simulator → Prompt templates: builds turns
core/llm/prompts.py - Evaluation coordinator → Rule and judge scoring: scores runs
core/orchestrator.py - Rule and judge scoring → Tester LLM client: judges qualitatively
core/evaluation/engine.py - Evaluation coordinator → Scenario JSON storage: loads scenarios
core/orchestrator.py - Agent LLM client → OpenRouter: sends requests
core/llm/client.py - Tester LLM client → OpenRouter: sends requests
core/llm/tester.py - Evaluation coordinator → Evaluation results: returns results
README.md - Scenario parser → Domain and result models: constructs
core/scenario/parser.py - Rule and judge scoring → Domain and result models: scores scenario models
core/evaluation/engine.py