# stanfordmimi/medval architecture

> Architecture diagram of the GitHub repository stanfordmimi/medval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/stanfordmimi/medval
- Repository: https://github.com/stanfordmimi/medval

## Overview

##### Project Type and Purpose
- MedVAL is a command‐line‐driven, full‐stack evaluation framework for medical‐text generation and validation.
- It orchestrates a pipeline that:
– Generates candidate medical text via a “generator” language‐model (API‐based or local)
– Validates the generated text’s factual consistency and safety via a “validator” language‐model
– Grades risk and outputs results

##### Main Components
a. CLI / Entry Point
– run.py: parses arguments, loads a YAML config (train or test), and invokes MedVAL.pipeline
b. Configuration Loader
– loads configs/test.yaml or configs/train.yaml
– exposes parameters: tasks, data split, model identifiers, API keys, fine‐tuning hyperparameters
c. Data Source
– MedVAL‐Bench dataset from HuggingFace (or custom CSV)
– Config‐driven loader inside pipeline.py
d. Pipeline Orchestrator
– medval/pipeline.py: controls the overall workflow
– Sequential stages: generation → validation → risk‐grading → result storage
e. Generator Module
– medval/generator.py: invokes the selected language model with task‐specific prompts
– Integrates with utils/prompts.py and task_prompts.json for prompt templates
f. Validator Module
– medval/validator.py: evaluates generated text for factual consistency and safety
– May call the same or different LM endpoints with specialized validation prompts
g. Utility Layer
– utils/: prompt definitions, JSON schemas, helper functions
h. External Model Services
– API‐based (OpenAI, Anthropic, Gemini, etc.)
– Local HuggingFace models (loaded from disk via Transformers)
– DSPy library for fine‐tuning
i. Results Storage
– results/ directory (zero-shot/finetune subfolders)
– CSV outputs named by model and dataset

##### Relationships and Data Flows
- run.py → Configuration Loader → Pipeline
- Pipeline reads data from the HF dataset or CSV (Data Source)
- Pipeline calls Generator → issues HTTP API calls or local-Transformer inferences → obtains generated text
- Pipeline passes generated text to Validator → issues API calls or local inferences → obtains validation scores & risk grades
- Pipeline writes annotated outputs to Results Storage

##### Architectural Patterns & Design Principles
- Modular separation of concerns: CLI, config loader, pipeline, generator, validator, utils
- Config‐driven architecture: behavior changes by swapping YAML files
- Self‐supervised validation loop: generator+validator chain

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Entry Point
    Run["run.py"]:::internal
    click Run "https://github.com/stanfordmimi/medval/blob/main/run.py"
    
    %% Configuration Layer
    subgraph "Configuration Layer"
        Loader["Configuration Loader"]:::internal
        TrainYAML["train.yaml"]:::internal
        TestYAML["test.yaml"]:::internal
    end
    click TrainYAML "https://github.com/stanfordmimi/medval/blob/main/configs/train.yaml"
    click TestYAML "https://github.com/stanfordmimi/medval/blob/main/configs/test.yaml"
    
    %% Core Pipeline
    subgraph "Core Pipeline"
        Pipeline["Pipeline Orchestrator\n(medval/pipeline.py)"]:::internal
        Generator["Generator Module\n(medval/generator.py)"]:::internal
        Validator["Validator Module\n(medval/validator.py)"]:::internal
    end
    click Pipeline "https://github.com/stanfordmimi/medval/blob/main/medval/pipeline.py"
    click Generator "https://github.com/stanfordmimi/medval/blob/main/medval/generator.py"
    click Validator "https://github.com/stanfordmimi/medval/blob/main/medval/validator.py"
    
    %% Utility Layer
    subgraph "Utility Layer"
        PromptsPy["prompts.py"]:::internal
        TaskPrompts["task_prompts.json"]:::internal
    end
    click PromptsPy "https://github.com/stanfordmimi/medval/blob/main/utils/prompts.py"
    click TaskPrompts "https://github.com/stanfordmimi/medval/blob/main/utils/task_prompts.json"
    
    %% Data Layer
    subgraph "Data Layer"
        DataSource["MedVAL-Bench Dataset"]:::data
        Results["Results Storage"]:::data
    end
    
    %% External Model Services
    subgraph "External Model Services"
        API["LM API (OpenAI, Anthropic, etc.)"]:::external
        Local["Local HF Models"]:::external
        DSPy["DSPy Library\n(fine-tuning)"]:::external
    end
    
    %% Documentation & Environment
    subgraph "Project Metadata"
        Env["env.yml"]:::docs
        README["README.md"]:::docs
        License["LICENSE"]:::docs
        Gitignore[".gitignore"]:::docs
    end
    click Env "https://github.com/stanfordmimi/medval/blob/main/env.yml"
    click README "https://github.com/stanfordmimi/medval/blob/main/README.md"
    click License "https://github.com/stanfordmimi/medval/tree/main/LICENSE"
    click Gitignore "https://github.com/stanfordmimi/medval/blob/main/.gitignore"
    
    %% Connections
    Run -->|parses args| Loader
    Loader -->|loads parameters| Pipeline
    Pipeline -->|reads data| DataSource
    Pipeline -->|"prompt + input text"| Generator
    Generator -->|"API call / inference"| API
    API -->|"generated text"| Generator
    Generator -->|"generated text"| Pipeline
    Pipeline -->|"validation prompt + text"| Validator
    Validator -->|"API call / inference"| API
    API -->|"validation scores"| Validator
    Validator -->|"scores & risk grades"| Pipeline
    Pipeline -->|"write results"| Results
    
    %% Pluggable Backends
    Generator -->|"local inference"| Local
    Local -->|"generated text"| Generator
    Validator -->|"local inference"| Local
    Local -->|"validation scores"| Validator
    
    %% Fine-tuning Flow
    Pipeline -->|"optional fine-tune"| DSPy
    DSPy -->|"fine-tuned model"| Pipeline
    
    %% Utils connections
    Generator --> PromptsPy
    Generator --> TaskPrompts
    Validator --> PromptsPy
    Validator --> TaskPrompts
    
    %% Styling
    classDef internal fill:#D6EAF8,stroke:#31708F,color:#000;
    classDef data fill:#D5F5E3,stroke:#27AE60,color:#000;
    classDef external fill:#FAD7A0,stroke:#D35400,color:#000;
    classDef docs fill:#F2F3F4,stroke:#ABB2B9,color:#000;
```
