# ziadisafouene/multiturn-agent-evaluator architecture

> Architecture diagram of the GitHub repository ziadisafouene/multiturn-agent-evaluator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ziadisafouene/multiturn-agent-evaluator
- Repository: https://github.com/ziadisafouene/multiturn-agent-evaluator

## Overview

multiturn-agent-eval is a framework for stress-testing multi-turn conversational agents. Users configure an evaluation through the CLI or Python API, supply scenarios or a natural-language specification, generate persona/language/mutation test cases, simulate conversations, and score results with deterministic checks and an optional LLM judge. The graph preserves the documented synthesis, multilingual, adversarial, and dual-scoring capabilities. Some orchestration and report-writing details are only partially excerpted, so uncertain internal links are omitted.

Main entry points: [`core/orchestrator.py`](https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/orchestrator.py) (Evaluation coordinator), [`core/simulation/simulator.py`](https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/simulation/simulator.py) (Conversation simulator), [`core/scenario/generator.py`](https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/scenario/generator.py) (Scenario synthesis), [`core/llm/tester.py`](https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/llm/tester.py) (Tester LLM client)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry and coordination"]
  node_cli["CLI<br/>[cli.py]"]
  node_orchestrator["Evaluation coordinator<br/>[orchestrator.py]"]
end

subgraph group_scenario["Scenario definition"]
  node_scenario_input["Scenario input"]
  node_scenario_generation["Scenario synthesis<br/>[generator.py]"]
  node_scenario_parser["Scenario parser<br/>[parser.py]"]
  node_scenario_validation["Scenario validation<br/>[validator.py]"]
  node_scenario_storage["Scenario JSON storage<br/>[storage.py]"]
  node_state_machine["State machine<br/>[state_machine.py]"]
end

subgraph group_cases["Test case design"]
  node_mutation["Mutation generator<br/>[generator.py]"]
  node_multilingual["Language resources<br/>[multilingual.py]"]
end

subgraph group_runtime["Conversation runtime"]
  node_simulator["Conversation simulator<br/>[simulator.py]"]
  node_agent["Agent adapter<br/>[agent.py]"]
  node_persona["Persona rewriting<br/>[engine.py]"]
  node_prompts["Prompt templates<br/>[prompts.py]"]
  node_agent_llm["Agent LLM client<br/>[client.py]"]
  node_tester_llm["Tester LLM client<br/>[tester.py]"]
end

subgraph group_scoring["Evaluation and models"]
  node_evaluation["Rule and judge scoring<br/>[engine.py]"]
  node_models["Domain and result models<br/>[models.py]"]
end

node_user(("Evaluator"))
node_openrouter{{"OpenRouter"}}
node_reports["Evaluation results"]

node_user -->|"runs"| node_cli
node_user -->|"configures"| node_orchestrator
node_cli -->|"invokes"| node_orchestrator
node_orchestrator -->|"accepts"| node_scenario_input
node_scenario_input -->|"loads JSON"| node_scenario_storage
node_scenario_storage -->|"parses"| node_scenario_parser
node_orchestrator -->|"requests synthesis"| node_scenario_generation
node_scenario_generation -->|"generates with"| node_tester_llm
node_scenario_generation -->|"parses output"| node_scenario_parser
node_scenario_generation -->|"validates"| node_scenario_validation
node_orchestrator -->|"generates cases"| node_mutation
node_orchestrator -->|"runs simulations"| node_simulator
node_simulator -->|"uses"| node_agent
node_agent -->|"requests responses"| node_agent_llm
node_simulator -->|"queries"| node_agent_llm
node_simulator -->|"rewrites user turns"| node_persona
node_persona -->|"generates rewrite"| node_tester_llm
node_persona -->|"builds prompt"| node_prompts
node_simulator -->|"builds turns"| node_prompts
node_orchestrator -->|"scores runs"| node_evaluation
node_evaluation -->|"judges qualitatively"| node_tester_llm
node_orchestrator -->|"loads scenarios"| node_scenario_storage
node_agent_llm -->|"sends requests"| node_openrouter
node_tester_llm -->|"sends requests"| node_openrouter
node_orchestrator -->|"returns results"| node_reports
node_scenario_parser -->|"constructs"| node_models
node_evaluation -->|"scores scenario models"| node_models

click node_cli "https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/cli.py"
click node_orchestrator "https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/orchestrator.py"
click node_scenario_generation "https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/scenario/generator.py"
click node_scenario_parser "https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/scenario/parser.py"
click node_scenario_validation "https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/scenario/validator.py"
click node_scenario_storage "https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/scenario/storage.py"
click node_state_machine "https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/scenario/state_machine.py"
click node_mutation "https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/mutation/generator.py"
click node_multilingual "https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/data/multilingual.py"
click node_simulator "https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/simulation/simulator.py"
click node_agent "https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/simulation/agent.py"
click node_persona "https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/persona/engine.py"
click node_prompts "https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/llm/prompts.py"
click node_agent_llm "https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/llm/client.py"
click node_tester_llm "https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/llm/tester.py"
click node_evaluation "https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/evaluation/engine.py"
click node_models "https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/utils/models.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_orchestrator toneBlue
class node_scenario_input,node_scenario_generation,node_scenario_parser,node_scenario_validation,node_scenario_storage,node_state_machine toneAmber
class node_mutation,node_multilingual,node_openrouter toneMint
class node_simulator,node_agent,node_persona,node_prompts,node_agent_llm,node_tester_llm toneRose
class node_evaluation,node_models,node_user,node_reports toneIndigo
```

## Components

### Entry and coordination

- [CLI](https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/cli.py): `core/cli.py`
- [Evaluation coordinator](https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/orchestrator.py): `core/orchestrator.py`

### Scenario definition

- **Scenario input**
- [Scenario synthesis](https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/scenario/generator.py): `core/scenario/generator.py`
- [Scenario parser](https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/scenario/parser.py): `core/scenario/parser.py`
- [Scenario validation](https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/scenario/validator.py): `core/scenario/validator.py`
- [Scenario JSON storage](https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/scenario/storage.py): `core/scenario/storage.py`
- [State machine](https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/scenario/state_machine.py): `core/scenario/state_machine.py`

### Test case design

- [Mutation generator](https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/mutation/generator.py): `core/mutation/generator.py`
- [Language resources](https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/data/multilingual.py): `core/data/multilingual.py`

### Conversation runtime

- [Conversation simulator](https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/simulation/simulator.py): `core/simulation/simulator.py`
- [Agent adapter](https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/simulation/agent.py): `core/simulation/agent.py`
- [Persona rewriting](https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/persona/engine.py): `core/persona/engine.py`
- [Prompt templates](https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/llm/prompts.py): `core/llm/prompts.py`
- [Agent LLM client](https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/llm/client.py): `core/llm/client.py`
- [Tester LLM client](https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/llm/tester.py): `core/llm/tester.py`

### Evaluation and models

- [Rule and judge scoring](https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/evaluation/engine.py): `core/evaluation/engine.py`
- [Domain and result models](https://github.com/ziadisafouene/multiturn-agent-evaluator/blob/main/core/utils/models.py): `core/utils/models.py`

### Other components

- **Evaluator**
- **OpenRouter**
- **Evaluation results**

## Connections

- Evaluator → CLI: runs (evidence: `README.md`)
- Evaluator → Evaluation coordinator: configures (evidence: `README.md`)
- CLI → Evaluation coordinator: invokes (evidence: `core/cli.py`)
- Evaluation coordinator → Scenario input: accepts (evidence: `README.md`)
- Scenario input → Scenario JSON storage: loads JSON (evidence: `core/scenario/storage.py`)
- Scenario JSON storage → Scenario parser: parses (evidence: `core/scenario/storage.py`)
- Evaluation coordinator → Scenario synthesis: requests synthesis (evidence: `core/orchestrator.py`)
- Scenario synthesis → Tester LLM client: generates with (evidence: `core/scenario/generator.py`)
- Scenario synthesis → Scenario parser: parses output (evidence: `core/scenario/generator.py`)
- Scenario synthesis → Scenario validation: validates (evidence: `core/scenario/generator.py`)
- Evaluation coordinator → Mutation generator: generates cases (evidence: `core/orchestrator.py`)
- Evaluation coordinator → Conversation simulator: runs simulations (evidence: `core/orchestrator.py`)
- Conversation simulator → Agent adapter: uses (evidence: `core/simulation/agent.py`)
- Agent adapter → Agent LLM client: requests responses (evidence: `core/simulation/agent.py`)
- Conversation simulator → Agent LLM client: queries (evidence: `core/simulation/simulator.py`)
- Conversation simulator → Persona rewriting: rewrites user turns (evidence: `core/simulation/simulator.py`)
- Persona rewriting → Tester LLM client: generates rewrite (evidence: `core/persona/engine.py`)
- Persona rewriting → Prompt templates: builds prompt (evidence: `core/persona/engine.py`)
- Conversation simulator → Prompt templates: builds turns (evidence: `core/llm/prompts.py`)
- Evaluation coordinator → Rule and judge scoring: scores runs (evidence: `core/orchestrator.py`)
- Rule and judge scoring → Tester LLM client: judges qualitatively (evidence: `core/evaluation/engine.py`)
- Evaluation coordinator → Scenario JSON storage: loads scenarios (evidence: `core/orchestrator.py`)
- Agent LLM client → OpenRouter: sends requests (evidence: `core/llm/client.py`)
- Tester LLM client → OpenRouter: sends requests (evidence: `core/llm/tester.py`)
- Evaluation coordinator → Evaluation results: returns results (evidence: `README.md`)
- Scenario parser → Domain and result models: constructs (evidence: `core/scenario/parser.py`)
- Rule and judge scoring → Domain and result models: scores scenario models (evidence: `core/evaluation/engine.py`)
