How almolsiaboelm-tech/medical-ai-evaluation-lab works
Medical AI Evaluation Lab is an experimental workflow for generating model responses to synthetic medical cases and evaluating them independently with a structured rubric. Its main path runs Day 4 cases through a MedGemma client, saves raw outputs, combines them with human-scored cases, and writes safety-aware evaluation results. A separate synthetic-case runner uses the same scoring engine. Scope and rubric documents define the evaluation boundaries and criteria. The graph follows documented workflows and sampled calls; unsampled details are omitted.
Main entry points: run_day4_evaluation.py Day 4 evaluation runner · run_medgemma_day4.py Generation runner · run_evaluation.py Synthetic evaluation runner · medgemma_client.py MedGemma client
Components 15
Evaluation definition
- Evaluation scope
scope/evaluation_scope_v1.md - Evaluation rubric
rubric/evaluation_rubric_v1.md
Model generation
- Day 4 cases
cases/day4_cases.json - Generation runner
src/run_medgemma_day4.py - MedGemma client
medgemma_client.py - Raw model outputs
results/model_outputs_day4.json
Scoring and results
- Scored Day 4 cases
cases/day4_scored_cases.json - Day 4 evaluation runner
src/run_day4_evaluation.py - Synthetic cases
cases/synthetic_cases_v1.json - Synthetic evaluation runner
src/run_evaluation.py - Scoring engine
src/evaluator.py - Day 4 results
results/evaluation_results_day4.json - Synthetic results
results/evaluation_results_v1.json
Other components
- Remote model endpoint
- Human reviewer
Connections 13
- Generation runner → Day 4 cases: reads cases
src/run_medgemma_day4.py - Generation runner → MedGemma client: requests generation
src/run_medgemma_day4.py - MedGemma client → Remote model endpoint: sends prompt
medgemma_client.py - Generation runner → Raw model outputs: writes outputs
src/run_medgemma_day4.py - Human reviewer → Evaluation rubric: reviews against
README.md - Human reviewer → Scored Day 4 cases: records scores
README.md - Day 4 evaluation runner → Scored Day 4 cases: reads scores
src/run_day4_evaluation.py - Day 4 evaluation runner → Raw model outputs: reads responses
src/run_day4_evaluation.py - Day 4 evaluation runner → Scoring engine: evaluates scores
src/run_day4_evaluation.py - Day 4 evaluation runner → Day 4 results: writes results
src/run_day4_evaluation.py - Synthetic evaluation runner → Synthetic cases: reads cases
src/run_evaluation.py - Synthetic evaluation runner → Scoring engine: evaluates scores
src/run_evaluation.py - Synthetic evaluation runner → Synthetic results: writes results
src/run_evaluation.py