# kimdanny/marketplace-eval architecture

> Architecture diagram of the GitHub repository kimdanny/marketplace-eval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kimdanny/marketplace-eval
- Repository: https://github.com/kimdanny/marketplace-eval

## Overview

Marketplace Eval simulates users choosing among competing generator agents whose retrieval workflows are routed through retrievers; a judge scores answers and feedback changes user preferences and routing behavior. The system builds the configured graph, advances simulation steps, and logs interactions. Users and question data can be configured or synthesized, while post-simulation tools compute market share and customer-retention metrics and produce plots. Relationships involving unsampled modules are included only where the README or source index supports them; uncertain wiring is omitted.

Main entry points: [`src/marketplace_eval/system/system.py`](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/system/system.py) (Simulation system), [`src/marketplace_eval/agents/generator_agent.py`](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/agents/generator_agent.py) (Generator and retrieval planner), [`src/marketplace_eval/system/user_data_utils.py`](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/system/user_data_utils.py) (User data utilities), [`src/marketplace_eval/post_simulation/cli.py`](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/post_simulation/cli.py) (Analysis CLI)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_simulation["Simulation runtime"]
  node_system["Simulation system<br/>[system.py]"]
  node_graphcore["Graph primitives<br/>[node.py]"]
  node_ionodes["Input/output nodes<br/>[io_nodes.py]"]
  node_links["Directed links<br/>[link.py]"]
  node_logger["Interaction logger"]
  node_types["Simulation records<br/>[types.py]"]
end

subgraph group_agents["Marketplace agents"]
  node_agentbase["Agent base<br/>[agent.py]"]
  node_generator["Generator and retrieval planner<br/>[generator_agent.py]"]
  node_router["Retriever router<br/>[router_agent.py]"]
  node_retriever["Retriever agents<br/>[retriever_agent.py]"]
  node_judge["LLM judge<br/>[llm_judge_agent.py]"]
  node_llm["LLM clients<br/>[llm_client.py]"]
  node_judgeprompt["Judge prompt"]
end

subgraph group_users["Users and questions"]
  node_population["User population<br/>[user_population.py]"]
  node_userprofiles["Profile generation"]
  node_userdata["User data utilities<br/>[user_data_utils.py]"]
  node_taxonomies["User taxonomies"]
  node_tunataxonomy["TUNA taxonomy<br/>[tuna_config.py]"]
  node_synthetic["Synthetic dataset generation"]
  node_userprompt["User simulation prompts"]
end

subgraph group_analysis["Post-simulation analysis"]
  node_marketshare["Market share metrics<br/>[market_share.py]"]
  node_retention["Retention metrics<br/>[crr.py]"]
  node_plot["Analysis plots<br/>[plot.py]"]
  node_analysiscli["Analysis CLI<br/>[cli.py]"]
end

node_operator(("Experiment operator"))
node_logfile["Simulation log CSV"]

node_operator -->|"configures and runs"| node_system
node_system -->|"samples users"| node_population
node_system -->|"builds graph"| node_generator
node_system -->|"builds graph"| node_router
node_system -->|"builds graph"| node_retriever
node_system -->|"creates judge"| node_judge
node_system -->|"connects nodes"| node_links
node_system -->|"logs steps"| node_logger
node_system -->|"creates profiles"| node_userprofiles
node_system -->|"loads user data"| node_userdata
node_system -->|"uses records"| node_types
node_generator -->|"requests routing"| node_router
node_generator -->|"invokes retrieval"| node_retriever
node_router -->|"selects from"| node_retriever
node_generator -->|"generates answers"| node_llm
node_judge -->|"requests evaluation"| node_llm
node_judgeprompt -.->|"guides evaluation"| node_judge
node_population -->|"samples profiles"| node_types
node_userprofiles -->|"creates profiles"| node_types
node_userdata -->|"uses taxonomy"| node_taxonomies
node_userdata -->|"uses prompts"| node_userprompt
node_synthetic -->|"formats questions"| node_userprompt
node_synthetic -->|"uses taxonomy"| node_taxonomies
node_system -->|"exports interactions"| node_logfile
node_operator -->|"runs analysis"| node_analysiscli
node_analysiscli -->|"computes metrics"| node_marketshare
node_analysiscli -->|"creates plots"| node_plot
node_logfile -->|"reads interaction log"| node_marketshare
node_logfile -->|"reads interaction log"| node_retention
node_marketshare -->|"supplies metrics"| node_plot

click node_system "https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/system/system.py"
click node_graphcore "https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/core/node.py"
click node_ionodes "https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/core/io_nodes.py"
click node_links "https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/core/link.py"
click node_logger "https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/system/simulation_logger.py"
click node_types "https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/system/types.py"
click node_agentbase "https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/agents/agent.py"
click node_generator "https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/agents/generator_agent.py"
click node_router "https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/agents/router_agent.py"
click node_retriever "https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/agents/retriever_agent.py"
click node_judge "https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/agents/llm_judge_agent.py"
click node_llm "https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/utils/llm_client.py"
click node_population "https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/system/user_population.py"
click node_userprofiles "https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/humans/profile_generator.py"
click node_userdata "https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/system/user_data_utils.py"
click node_taxonomies "https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/humans/datamorgana_config.py"
click node_tunataxonomy "https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/humans/tuna_config.py"
click node_synthetic "https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/humans/synthetic_dataset_generation.py"
click node_userprompt "https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/prompts/user_simulation_prompt.py"
click node_judgeprompt "https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/prompts/simple_qa_judge_prompt.py"
click node_marketshare "https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/post_simulation/market_share.py"
click node_retention "https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/post_simulation/crr.py"
click node_plot "https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/post_simulation/plot.py"
click node_analysiscli "https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/post_simulation/cli.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_system,node_graphcore,node_ionodes,node_links,node_logger,node_types toneBlue
class node_agentbase,node_generator,node_router,node_retriever,node_judge,node_llm,node_judgeprompt toneAmber
class node_population,node_userprofiles,node_userdata,node_taxonomies,node_tunataxonomy,node_synthetic,node_userprompt toneMint
class node_marketshare,node_retention,node_plot,node_analysiscli toneRose
class node_operator,node_logfile toneIndigo
```

## Components

### Simulation runtime

- [Simulation system](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/system/system.py): `src/marketplace_eval/system/system.py`
- [Graph primitives](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/core/node.py): `src/marketplace_eval/core/node.py`
- [Input/output nodes](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/core/io_nodes.py): `src/marketplace_eval/core/io_nodes.py`
- [Directed links](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/core/link.py): `src/marketplace_eval/core/link.py`
- [Interaction logger](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/system/simulation_logger.py): `src/marketplace_eval/system/simulation_logger.py`
- [Simulation records](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/system/types.py): `src/marketplace_eval/system/types.py`

### Marketplace agents

- [Agent base](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/agents/agent.py): `src/marketplace_eval/agents/agent.py`
- [Generator and retrieval planner](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/agents/generator_agent.py): `src/marketplace_eval/agents/generator_agent.py`
- [Retriever router](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/agents/router_agent.py): `src/marketplace_eval/agents/router_agent.py`
- [Retriever agents](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/agents/retriever_agent.py): `src/marketplace_eval/agents/retriever_agent.py`
- [LLM judge](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/agents/llm_judge_agent.py): `src/marketplace_eval/agents/llm_judge_agent.py`
- [LLM clients](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/utils/llm_client.py): `src/marketplace_eval/utils/llm_client.py`
- [Judge prompt](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/prompts/simple_qa_judge_prompt.py): `src/marketplace_eval/prompts/simple_qa_judge_prompt.py`

### Users and questions

- [User population](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/system/user_population.py): `src/marketplace_eval/system/user_population.py`
- [Profile generation](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/humans/profile_generator.py): `src/marketplace_eval/humans/profile_generator.py`
- [User data utilities](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/system/user_data_utils.py): `src/marketplace_eval/system/user_data_utils.py`
- [User taxonomies](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/humans/datamorgana_config.py): `src/marketplace_eval/humans/datamorgana_config.py`
- [TUNA taxonomy](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/humans/tuna_config.py): `src/marketplace_eval/humans/tuna_config.py`
- [Synthetic dataset generation](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/humans/synthetic_dataset_generation.py): `src/marketplace_eval/humans/synthetic_dataset_generation.py`
- [User simulation prompts](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/prompts/user_simulation_prompt.py): `src/marketplace_eval/prompts/user_simulation_prompt.py`

### Post-simulation analysis

- [Market share metrics](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/post_simulation/market_share.py): `src/marketplace_eval/post_simulation/market_share.py`
- [Retention metrics](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/post_simulation/crr.py): `src/marketplace_eval/post_simulation/crr.py`
- [Analysis plots](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/post_simulation/plot.py): `src/marketplace_eval/post_simulation/plot.py`
- [Analysis CLI](https://github.com/kimdanny/marketplace-eval/blob/main/src/marketplace_eval/post_simulation/cli.py): `src/marketplace_eval/post_simulation/cli.py`

### Other components

- **Experiment operator**
- **Simulation log CSV**

## Connections

- Experiment operator → Simulation system: configures and runs (evidence: `README.md`)
- Simulation system → User population: samples users (evidence: `src/marketplace_eval/system/system.py`)
- Simulation system → Generator and retrieval planner: builds graph (evidence: `src/marketplace_eval/system/system.py`)
- Simulation system → Retriever router: builds graph (evidence: `src/marketplace_eval/system/system.py`)
- Simulation system → Retriever agents: builds graph (evidence: `src/marketplace_eval/system/system.py`)
- Simulation system → LLM judge: creates judge (evidence: `src/marketplace_eval/system/system.py`)
- Simulation system → Directed links: connects nodes (evidence: `src/marketplace_eval/system/system.py`)
- Simulation system → Interaction logger: logs steps (evidence: `src/marketplace_eval/system/system.py`)
- Simulation system → Profile generation: creates profiles (evidence: `src/marketplace_eval/system/system.py`)
- Simulation system → User data utilities: loads user data (evidence: `src/marketplace_eval/system/system.py`)
- Simulation system → Simulation records: uses records (evidence: `src/marketplace_eval/system/system.py`)
- Generator and retrieval planner → Retriever router: requests routing (evidence: `src/marketplace_eval/agents/generator_agent.py`)
- Generator and retrieval planner → Retriever agents: invokes retrieval (evidence: `src/marketplace_eval/agents/generator_agent.py`)
- Retriever router → Retriever agents: selects from (evidence: `src/marketplace_eval/agents/router_agent.py`)
- Generator and retrieval planner → LLM clients: generates answers (evidence: `src/marketplace_eval/agents/generator_agent.py`)
- LLM judge → LLM clients: requests evaluation (evidence: `src/marketplace_eval/agents/llm_judge_agent.py`)
- Judge prompt → LLM judge: guides evaluation
- User population → Simulation records: samples profiles (evidence: `src/marketplace_eval/system/user_population.py`)
- Profile generation → Simulation records: creates profiles (evidence: `src/marketplace_eval/humans/profile_generator.py`)
- User data utilities → User taxonomies: uses taxonomy (evidence: `src/marketplace_eval/system/user_data_utils.py`)
- User data utilities → User simulation prompts: uses prompts (evidence: `src/marketplace_eval/system/user_data_utils.py`)
- Synthetic dataset generation → User simulation prompts: formats questions (evidence: `src/marketplace_eval/humans/synthetic_dataset_generation.py`)
- Synthetic dataset generation → User taxonomies: uses taxonomy (evidence: `src/marketplace_eval/humans/synthetic_dataset_generation.py`)
- Simulation system → Simulation log CSV: exports interactions (evidence: `README.md`)
- Experiment operator → Analysis CLI: runs analysis (evidence: `README.md`)
- Analysis CLI → Market share metrics: computes metrics (evidence: `src/marketplace_eval/post_simulation/cli.py`)
- Analysis CLI → Analysis plots: creates plots (evidence: `src/marketplace_eval/post_simulation/cli.py`)
- Simulation log CSV → Market share metrics: reads interaction log (evidence: `src/marketplace_eval/post_simulation/market_share.py`)
- Simulation log CSV → Retention metrics: reads interaction log (evidence: `README.md`)
- Market share metrics → Analysis plots: supplies metrics (evidence: `src/marketplace_eval/post_simulation/plot.py`)
