How nii-nlp/med-eval works
Project Type and Purpose
• Med-Eval is an evaluation framework that supports multiple biomedical tasks (MCQA, MT, NER, Document Classification, STS) using various prompt templates.
• It is designed as a flexible alternative to the lm-evaluation-harness, particularly catering to the evaluation challenges of medical language models (e.g., diverse templates, multiple GPUs, local datasets).
• It is structured as a submodule that integrates with a higher-level repository (JMed-LLM) and leverages PyTorch for multi-GPU Distributed Data Parallel (DDP) evaluation.
File Structure
• Key files at the top level include several README files (for documentation in multiple languages), configuration (config_file.py), and requirements for dependency management.
• Directories:
• data_loaders/ and data_utils.py: Responsible for loading and preprocessing datasets.
• tasks/: Contains task-specific modules (e.g., mcqa.py, mt.py, ner.py, nli.py) that define dataset loading and formatting for evaluation.
• templates/: Houses prompt templates (e.g., for MCQA, MT, NER, etc.) that define the textual structure for each evaluation type.
• Scripts: Multiple directories (scripts/batch, scripts/evaluation, scripts/kate, scripts/run_all_template) provide bash scripts to launch batch evaluations or to process custom data.
• The main evaluation driver files such as evaluate_mcqa.py, evaluate_mt.py, etc., and pipeline.py which ties together model loading, data processing, and evaluation logic.
• The structure indicates a modular architecture where:
• The evaluation core (the “EvaluationPipeline” in pipeline.py) orchestrates the workflow.
• Task-specific logic (located in tasks/) and prompt/formats (located in templates/) are separated.
• Various shell scripts enable different evaluation modes (batch vs. single-run).
Main Components
a. Configuration & Setup:
• config_file.py that defines settings (e.g., dataset paths).
• Environment setup instructions (e.g., dependencies via requirements.txt, PyTorch + CUDA).
b. Evaluation Pipeline:
• pipeline.py which initializes the evaluation environment (supporting both single and multi-GPU setups using PyTorch DDP).
• Model Loading: Integrates model and tokenizer loading (support any HF Hugging Face model).
c. Data Handling:
• data_loaders/ and data_utils.py: Modules for loading datasets in the expected format.
• External data source: Datasets downloaded via Git LFS/Hugging Face (e.g., JMedBench).
d. Task Management:
• tasks/ directory, where each file (mcqa.py, mt.py, ner.py, nli.py) encapsulates the logic for a particular evaluation task.
• Task-specific dataset conversion into a defined dataclass (e.g., MCQASample) used by the evaluation pipeline.
e. Prompt Templates:
• templates/ directory which hosts various templates (for example, mcqa.py, mt.py, etc.) controlling how questions/prompts are formatted.
f. Evaluation Scripts:
• Dedicated scripts (e.g., evaluate_mcqa.py, evaluate_mt.py, evaluate_ner.py, etc.) that act as entry points for running evaluations.
• Batch and evaluation shell scripts (found in scripts/*) which automate the execution of the submodule for different tasks.
g. Auxiliary Utilities:
• Additional modules like tool_utils.py and miscellaneous scripts to support tasks such as data conversion (e.g., the kate directory scripts).
Relationships and Interactions
• At the top-level, a user or automation script calls one of the evaluation scripts (e.g., evaluate_mcqa.py).
• The evaluation script instantiates the EvaluationPipeline component which triggers:
• Environment and GPU setup.
• Model and tokenizer loading.
• Dataset loading via functions in data_loaders/ and tasks/ corresponding to the chosen task.
• Prompt template selection from the templates/ module to format inputs.
• After processing, the pipeline collects computed losses from different GPUs and merges the results.
• The shell scripts (e.g., scripts/evaluation/*) wrap these processes to enable batch evaluations, making sure the proper parameters (e.g., model name, task, template, batch size) are passed.
• External dataset sources are managed via Git LFS/Hugging Face and their paths can be updated in config_file.py.