How talhaahmed86402/medibuddy works
MediBuddy-Bench is an offline, deterministic benchmark for agents assisting an international resident in Korea. Its main workflow is to generate synthetic scenarios, expose task cards and tools to an agent, run the interaction, score task-specific answers and safety failures, and record results for comparison. The product covers hospital choice, triage, interpretation, visit summaries, cross-visit memory, and full journeys; it also supports remote agents through A2A and model sweeps. Edges reflect README statements and SOURCE INDEX references; unsampled implementation details are not inferred.
Main entry points: runner.py Episode runner · build.py Dataset builder · a2a_green.py A2A assessment flow · choice.py Hospital scenarios
Components 25
Scenario generation
- Dataset builder
medibuddy_bench/gen/build.py - Hospital scenarios
medibuddy_bench/gen/choice.py - Encounter scenarios
medibuddy_bench/gen/encounters.py - Triage scenarios
medibuddy_bench/gen/triage.py - Journey scenarios
medibuddy_bench/gen/journey.py - Scenario phrasing
medibuddy_bench/gen/phrasing.py
Agent assessment
- Task cards
medibuddy_bench/taskcards.py - Agent tools
medibuddy_bench/tools.py - Episode runner
medibuddy_bench/runner.py
Task domains
- Simulated world
medibuddy_bench/world.py - World knowledge
medibuddy_bench/knowledge.py
Scoring and results
- Task scoring
medibuddy_bench/scoring/choice.py - Fact checking
medibuddy_bench/scoring/facts.py - Score aggregation
medibuddy_bench/scoring/aggregate.py - Run records
medibuddy_bench/results.py - Leaderboard
medibuddy_bench/leaderboard.py
Agent integrations
- Benchmark agent
medibuddy_bench/agents/base.py - Reference agents
medibuddy_bench/agents/baselines.py - Remote agent protocol
medibuddy_bench/protocol.py - OpenAI-compatible agent
medibuddy_bench/agents/llm_openai_compat.py - A2A assessment service
medibuddy_bench/a2a/green_server.py - A2A assessment flow
medibuddy_bench/a2a_green.py
Other components
- Benchmark user
- Model endpoint
- External A2A agent
Connections 31
- Benchmark user → Dataset builder: builds data
- Dataset builder → Hospital scenarios: generates
medibuddy_bench/gen/build.py - Dataset builder → Encounter scenarios: generates
medibuddy_bench/gen/build.py - Dataset builder → Triage scenarios: generates
medibuddy_bench/gen/build.py - Dataset builder → Journey scenarios: generates
medibuddy_bench/gen/build.py - Dataset builder → Simulated world: uses
medibuddy_bench/gen/build.py - Hospital scenarios → Scenario phrasing: uses
medibuddy_bench/gen/choice.py - Hospital scenarios → Simulated world: checks constraints
medibuddy_bench/gen/choice.py - Encounter scenarios → World knowledge: uses facts
medibuddy_bench/gen/encounters.py - Triage scenarios → World knowledge: uses facts
medibuddy_bench/gen/triage.py - Journey scenarios → Hospital scenarios: combines scenarios
medibuddy_bench/gen/journey.py - Journey scenarios → Encounter scenarios: combines scenarios
medibuddy_bench/gen/journey.py - Episode runner → Task cards: builds cards
medibuddy_bench/runner.py - Episode runner → Agent tools: provides tools
medibuddy_bench/runner.py - Episode runner → Simulated world: loads world
medibuddy_bench/runner.py - Episode runner → Benchmark agent: starts session
medibuddy_bench/runner.py - Reference agents → Benchmark agent: implements
medibuddy_bench/agents/baselines.py - Episode runner → Task scoring: scores tasks
medibuddy_bench/runner.py - Episode runner → Fact checking: checks facts
medibuddy_bench/runner.py - Task scoring → Fact checking: uses checkers
- Benchmark user → Episode runner: runs assessment
- Model endpoint → OpenAI-compatible agent: serves completions
medibuddy_bench/agents/llm_openai_compat.py - OpenAI-compatible agent → Remote agent protocol: adapts replies
medibuddy_bench/agents/llm_openai_compat.py - External A2A agent → A2A assessment service: sends assessment
- A2A assessment service → A2A assessment flow: dispatches assessment
medibuddy_bench/a2a/green_server.py - A2A assessment flow → Episode runner: runs episodes
medibuddy_bench/a2a_green.py - A2A assessment flow → Remote agent protocol: uses protocol
medibuddy_bench/a2a_green.py - A2A assessment flow → Score aggregation: aggregates scores
medibuddy_bench/a2a_green.py - Episode runner → Run records: records runs
medibuddy_bench/execute.py - Run records → Score aggregation: supplies episodes
medibuddy_bench/results.py - Leaderboard → Run records: reads records
medibuddy_bench/leaderboard.py