# nii-nlp/med-eval architecture

> Architecture diagram of the GitHub repository nii-nlp/med-eval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nii-nlp/med-eval
- Repository: https://github.com/nii-nlp/med-eval

## Overview

##### Project Type and Purpose
- Med-Eval is an evaluation framework that supports multiple biomedical tasks (MCQA, MT, NER, Document Classification, STS) using various prompt templates.
- It is designed as a flexible alternative to the lm-evaluation-harness, particularly catering to the evaluation challenges of medical language models (e.g., diverse templates, multiple GPUs, local datasets).
- It is structured as a submodule that integrates with a higher-level repository (JMed-LLM) and leverages PyTorch for multi-GPU Distributed Data Parallel (DDP) evaluation.

##### File Structure
- Key files at the top level include several README files (for documentation in multiple languages), configuration (config_file.py), and requirements for dependency management.
- Directories:
- data_loaders/ and data_utils.py: Responsible for loading and preprocessing datasets.
- tasks/: Contains task-specific modules (e.g., mcqa.py, mt.py, ner.py, nli.py) that define dataset loading and formatting for evaluation.
- templates/: Houses prompt templates (e.g., for MCQA, MT, NER, etc.) that define the textual structure for each evaluation type.
- Scripts: Multiple directories (scripts/batch, scripts/evaluation, scripts/kate, scripts/run_all_template) provide bash scripts to launch batch evaluations or to process custom data.
- The main evaluation driver files such as evaluate_mcqa.py, evaluate_mt.py, etc., and pipeline.py which ties together model loading, data processing, and evaluation logic.
- The structure indicates a modular architecture where:
- The evaluation core (the “EvaluationPipeline” in pipeline.py) orchestrates the workflow.
- Task-specific logic (located in tasks/) and prompt/formats (located in templates/) are separated.
- Various shell scripts enable different evaluation modes (batch vs. single-run).

##### Main Components
a. Configuration & Setup:
- config_file.py that defines settings (e.g., dataset paths).
- Environment setup instructions (e.g., dependencies via requirements.txt, PyTorch + CUDA).
b. Evaluation Pipeline:
- pipeline.py which initializes the evaluation environment (supporting both single and multi-GPU setups using PyTorch DDP).
- Model Loading: Integrates model and tokenizer loading (support any HF Hugging Face model).
c. Data Handling:
- data_loaders/ and data_utils.py: Modules for loading datasets in the expected format.
- External data source: Datasets downloaded via Git LFS/Hugging Face (e.g., JMedBench).
d. Task Management:
- tasks/ directory, where each file (mcqa.py, mt.py, ner.py, nli.py) encapsulates the logic for a particular evaluation task.
- Task-specific dataset conversion into a defined dataclass (e.g., MCQASample) used by the evaluation pipeline.
e. Prompt Templates:
- templates/ directory which hosts various templates (for example, mcqa.py, mt.py, etc.) controlling how questions/prompts are formatted.
f. Evaluation Scripts:
- Dedicated scripts (e.g., evaluate_mcqa.py, evaluate_mt.py, evaluate_ner.py, etc.) that act as entry points for running evaluations.
- Batch and evaluation shell scripts (found in scripts/*) which automate the execution of the submodule for different tasks.
g. Auxiliary Utilities:
- Additional modules like tool_utils.py and miscellaneous scripts to support tasks such as data conversion (e.g., the kate directory scripts).

##### Relationships and Interactions
- At the top-level, a user or automation script calls one of the evaluation scripts (e.g., evaluate_mcqa.py).
- The evaluation script instantiates the EvaluationPipeline component which triggers:
- Environment and GPU setup.
- Model and tokenizer loading.
- Dataset loading via functions in data_loaders/ and tasks/ corresponding to the chosen task.
- Prompt template selection from the templates/ module to format inputs.
- After processing, the pipeline collects computed losses from different GPUs and merges the results.
- The shell scripts (e.g., scripts/evaluation/*) wrap these processes to enable batch evaluations, making sure the proper parameters (e.g., model name, task, template, batch size) are passed.
- External dataset sources are managed via Git LFS/Hugging Face and their paths can be updated in config_file.py.

## Diagram (Mermaid)

```mermaid
graph TD
    subgraph "Configuration & Setup"
        config["config_file.py"]:::config
        req["requirements.txt"]:::config
    end

    subgraph "Evaluation Pipeline"
        pipe["Evaluation Pipeline (pipeline.py)"]:::core
    end

    subgraph "Data Handling"
        dl["Data Loaders (data_loaders/)"]:::data
        du["Data Utils (data_utils.py)"]:::data
    end

    subgraph "Task Management"
        tasks["Tasks Modules (tasks/)"]:::task
    end

    subgraph "Prompt Templates"
        templates["Templates (templates/)"]:::template
    end

    subgraph "Evaluation Scripts"
        evals["Evaluation Scripts (evaluate_*.py)"]:::script
        shell["Shell Scripts (scripts/)"]:::script
    end

    subgraph "Auxiliary Utilities"
        tool["Tool Utils (tool_utils.py)"]:::aux
        kate["Kate Utilities (kate/)"]:::aux
    end

    external["External Data Sources"]:::external
    user["User/Automation"]:::user
    output["User Output"]:::output

    %% Flow Connections
    user -->|"initiate"| evals
    evals -->|"calls"| pipe
    pipe -->|"loads config from"| config
    pipe -->|"loads config from"| req
    pipe -->|"loads data via"| dl
    pipe -->|"loads data via"| du
    external -->|"provides data to"| dl
    external -->|"provides data to"| du
    dl -->|"preprocesses for"| tasks
    du -->|"assists"| tasks
    templates -->|"formats input for"| tasks
    tasks -->|"evaluates and returns"| pipe
    pipe -->|"outputs results to"| output
    evals -->|"triggers"| shell
    shell -->|"batch execution of"| pipe
    tasks -->|"utilizes"| tool
    tool -->|"supports"| kate

    %% Click Events
    click config "https://github.com/nii-nlp/med-eval/blob/master/config_file.py"
    click req "https://github.com/nii-nlp/med-eval/blob/master/requirements.txt"
    click pipe "https://github.com/nii-nlp/med-eval/blob/master/pipeline.py"
    click dl "https://github.com/nii-nlp/med-eval/tree/master/data_loaders/"
    click du "https://github.com/nii-nlp/med-eval/blob/master/data_utils.py"
    click tasks "https://github.com/nii-nlp/med-eval/tree/master/tasks/"
    click templates "https://github.com/nii-nlp/med-eval/tree/master/templates/"
    click evals "https://github.com/nii-nlp/med-eval/blob/master/evaluate_mcqa.py"
    click shell "https://github.com/nii-nlp/med-eval/tree/master/scripts/"
    click tool "https://github.com/nii-nlp/med-eval/blob/master/tool_utils.py"
    click kate "https://github.com/nii-nlp/med-eval/tree/master/kate/"

    %% Styles
    classDef config fill:#f9c,stroke:#333,stroke-width:2px;
    classDef core fill:#bbf,stroke:#333,stroke-width:2px;
    classDef data fill:#cfc,stroke:#333,stroke-width:2px;
    classDef task fill:#ffc,stroke:#333,stroke-width:2px;
    classDef template fill:#fcf,stroke:#333,stroke-width:2px;
    classDef script fill:#cff,stroke:#333,stroke-width:2px;
    classDef aux fill:#fdd,stroke:#333,stroke-width:2px;
    classDef external fill:#f99,stroke:#333,stroke-width:2px;
    classDef user fill:#ddd,stroke:#333,stroke-width:2px;
    classDef output fill:#ddd,stroke:#333,stroke-width:2px;
```
