How kimdanny/marketplace-eval works
Marketplace Eval simulates users choosing among competing generator agents whose retrieval workflows are routed through retrievers; a judge scores answers and feedback changes user preferences and routing behavior. The system builds the configured graph, advances simulation steps, and logs interactions. Users and question data can be configured or synthesized, while post-simulation tools compute market share and customer-retention metrics and produce plots. Relationships involving unsampled modules are included only where the README or source index supports them; uncertain wiring is omitted.
Main entry points: system.py Simulation system · generator_agent.py Generator and retrieval planner · user_data_utils.py User data utilities · cli.py Analysis CLI
Components 26
Simulation runtime
- Simulation system
src/marketplace_eval/system/system.py - Graph primitives
src/marketplace_eval/core/node.py - Input/output nodes
src/marketplace_eval/core/io_nodes.py - Directed links
src/marketplace_eval/core/link.py - Interaction logger
src/marketplace_eval/system/simulation_logger.py - Simulation records
src/marketplace_eval/system/types.py
Marketplace agents
- Agent base
src/marketplace_eval/agents/agent.py - Generator and retrieval planner
src/marketplace_eval/agents/generator_agent.py - Retriever router
src/marketplace_eval/agents/router_agent.py - Retriever agents
src/marketplace_eval/agents/retriever_agent.py - LLM judge
src/marketplace_eval/agents/llm_judge_agent.py - LLM clients
src/marketplace_eval/utils/llm_client.py - Judge prompt
src/marketplace_eval/prompts/simple_qa_judge_prompt.py
Users and questions
- User population
src/marketplace_eval/system/user_population.py - Profile generation
src/marketplace_eval/humans/profile_generator.py - User data utilities
src/marketplace_eval/system/user_data_utils.py - User taxonomies
src/marketplace_eval/humans/datamorgana_config.py - TUNA taxonomy
src/marketplace_eval/humans/tuna_config.py - Synthetic dataset generation
src/marketplace_eval/humans/synthetic_dataset_generation.py - User simulation prompts
src/marketplace_eval/prompts/user_simulation_prompt.py
Post-simulation analysis
- Market share metrics
src/marketplace_eval/post_simulation/market_share.py - Retention metrics
src/marketplace_eval/post_simulation/crr.py - Analysis plots
src/marketplace_eval/post_simulation/plot.py - Analysis CLI
src/marketplace_eval/post_simulation/cli.py
Other components
- Experiment operator
- Simulation log CSV
Connections 30
- Experiment operator → Simulation system: configures and runs
README.md - Simulation system → User population: samples users
src/marketplace_eval/system/system.py - Simulation system → Generator and retrieval planner: builds graph
src/marketplace_eval/system/system.py - Simulation system → Retriever router: builds graph
src/marketplace_eval/system/system.py - Simulation system → Retriever agents: builds graph
src/marketplace_eval/system/system.py - Simulation system → LLM judge: creates judge
src/marketplace_eval/system/system.py - Simulation system → Directed links: connects nodes
src/marketplace_eval/system/system.py - Simulation system → Interaction logger: logs steps
src/marketplace_eval/system/system.py - Simulation system → Profile generation: creates profiles
src/marketplace_eval/system/system.py - Simulation system → User data utilities: loads user data
src/marketplace_eval/system/system.py - Simulation system → Simulation records: uses records
src/marketplace_eval/system/system.py - Generator and retrieval planner → Retriever router: requests routing
src/marketplace_eval/agents/generator_agent.py - Generator and retrieval planner → Retriever agents: invokes retrieval
src/marketplace_eval/agents/generator_agent.py - Retriever router → Retriever agents: selects from
src/marketplace_eval/agents/router_agent.py - Generator and retrieval planner → LLM clients: generates answers
src/marketplace_eval/agents/generator_agent.py - LLM judge → LLM clients: requests evaluation
src/marketplace_eval/agents/llm_judge_agent.py - Judge prompt → LLM judge: guides evaluation
- User population → Simulation records: samples profiles
src/marketplace_eval/system/user_population.py - Profile generation → Simulation records: creates profiles
src/marketplace_eval/humans/profile_generator.py - User data utilities → User taxonomies: uses taxonomy
src/marketplace_eval/system/user_data_utils.py - User data utilities → User simulation prompts: uses prompts
src/marketplace_eval/system/user_data_utils.py - Synthetic dataset generation → User simulation prompts: formats questions
src/marketplace_eval/humans/synthetic_dataset_generation.py - Synthetic dataset generation → User taxonomies: uses taxonomy
src/marketplace_eval/humans/synthetic_dataset_generation.py - Simulation system → Simulation log CSV: exports interactions
README.md - Experiment operator → Analysis CLI: runs analysis
README.md - Analysis CLI → Market share metrics: computes metrics
src/marketplace_eval/post_simulation/cli.py - Analysis CLI → Analysis plots: creates plots
src/marketplace_eval/post_simulation/cli.py - Simulation log CSV → Market share metrics: reads interaction log
src/marketplace_eval/post_simulation/market_share.py - Simulation log CSV → Retention metrics: reads interaction log
README.md - Market share metrics → Analysis plots: supplies metrics
src/marketplace_eval/post_simulation/plot.py