How kelkalot/normedqa works
Project Type and Purpose
• NorMedQA is not a web app or microservice suite but an open-source benchmarking pipeline for evaluating large language models (LLMs) on Norwegian medical question answering.
• It provides:
• A curated dataset (JSON files)
• Data split and preprocessing scripts (Python & Jupyter notebook)
• Configuration for the EleutherAI lm-evaluation-harness
• Metrics reporting (exact match, ROUGE variants)
File Structure Analysis
• Top level
• README.md – project description, usage, citation
• code/
• norwegian_medical_qa.py – Python script driving data split, harness run
• norwegian_medical_qa.ipynb – interactive notebook for Colab or local use
• data/
• norwegian_medical_qa_v2.json – full dataset
• norwegian_medical_qa_v2_txt_only.json – text-only subset
Key Architecture Components
a. Data Source & Storage
• External: Zenodo (dataset hosting)
• Local: data/*.json files after download
b. Data Split & Preprocessor
• Python code/notebook modules that read JSON, produce train/test splits
• Handles “wrong_answers” parsing for multiple-choice extension
c. Evaluation Harness
• lm-evaluation-harness library (external dependency)
• Configuration layer mapping NorMedQA to harness’s Task API
d. LLM Model API
• External service (e.g., OpenAI, Google Gemma) invoked by the harness
e. Metrics & Reporting
• exact_match, rouge1/2/L/Lsum computed by the harness
• Output saved/logged by script or notebook
f. Orchestration
• Jupyter notebook for guided setup & execution
• CLI Python script for automated runs in CI or local shell
Relationships & Data Flow
1. User/CI triggers the notebook or python script
2. Script downloads (or reads) JSON dataset from data/
3. Preprocessor component splits and cleans data
4. Harness component reads preprocessed data, invokes LLM API for each question
5. LLM returns answers; harness computes metrics
6. Metrics and raw predictions output to console, log files, or notebook cells
Architectural Patterns & Principles
• Modular pipeline: clear separation between Data Ingestion, Processing, Evaluation, Reporting
• Extensibility: “wrong_answers” field enables future multiple-choice testing
• Leverage existing libraries (Don’t reinvent the harness; plug in NorMedQA as a new Task)
• Reproducibility: JSON-based config, notebook walkthrough, script automation
• Clear, consistent labels on every component
• Local vs external
• Data vs compute vs storage
• Title and version annotation (e.g., “NorMedQA v2 Benchmark Architecture”)