How stanfordmimi/medval works
Project Type and Purpose
• MedVAL is a command‐line‐driven, full‐stack evaluation framework for medical‐text generation and validation.
• It orchestrates a pipeline that:
– Generates candidate medical text via a “generator” language‐model (API‐based or local)
– Validates the generated text’s factual consistency and safety via a “validator” language‐model
– Grades risk and outputs results
Main Components
a. CLI / Entry Point
– run.py: parses arguments, loads a YAML config (train or test), and invokes MedVAL.pipeline
b. Configuration Loader
– loads configs/test.yaml or configs/train.yaml
– exposes parameters: tasks, data split, model identifiers, API keys, fine‐tuning hyperparameters
c. Data Source
– MedVAL‐Bench dataset from HuggingFace (or custom CSV)
– Config‐driven loader inside pipeline.py
d. Pipeline Orchestrator
– medval/pipeline.py: controls the overall workflow
– Sequential stages: generation → validation → risk‐grading → result storage
e. Generator Module
– medval/generator.py: invokes the selected language model with task‐specific prompts
– Integrates with utils/prompts.py and task_prompts.json for prompt templates
f. Validator Module
– medval/validator.py: evaluates generated text for factual consistency and safety
– May call the same or different LM endpoints with specialized validation prompts
g. Utility Layer
– utils/: prompt definitions, JSON schemas, helper functions
h. External Model Services
– API‐based (OpenAI, Anthropic, Gemini, etc.)
– Local HuggingFace models (loaded from disk via Transformers)
– DSPy library for fine‐tuning
i. Results Storage
– results/ directory (zero-shot/finetune subfolders)
– CSV outputs named by model and dataset
Relationships and Data Flows
• run.py → Configuration Loader → Pipeline
• Pipeline reads data from the HF dataset or CSV (Data Source)
• Pipeline calls Generator → issues HTTP API calls or local-Transformer inferences → obtains generated text
• Pipeline passes generated text to Validator → issues API calls or local inferences → obtains validation scores & risk grades
• Pipeline writes annotated outputs to Results Storage
Architectural Patterns & Design Principles
• Modular separation of concerns: CLI, config loader, pipeline, generator, validator, utils
• Config‐driven architecture: behavior changes by swapping YAML files
• Self‐supervised validation loop: generator+validator chain