# kelkalot/normedqa architecture

> Architecture diagram of the GitHub repository kelkalot/normedqa, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kelkalot/normedqa
- Repository: https://github.com/kelkalot/normedqa

## Overview

##### Project Type and Purpose
- NorMedQA is not a web app or microservice suite but an open-source benchmarking pipeline for evaluating large language models (LLMs) on Norwegian medical question answering.
- It provides:
- A curated dataset (JSON files)
- Data split and preprocessing scripts (Python & Jupyter notebook)
- Configuration for the EleutherAI lm-evaluation-harness
- Metrics reporting (exact match, ROUGE variants)

##### File Structure Analysis
- Top level
- README.md – project description, usage, citation
- code/
- norwegian_medical_qa.py – Python script driving data split, harness run
- norwegian_medical_qa.ipynb – interactive notebook for Colab or local use
- data/
- norwegian_medical_qa_v2.json – full dataset
- norwegian_medical_qa_v2_txt_only.json – text-only subset

##### Key Architecture Components
a. Data Source & Storage
- External: Zenodo (dataset hosting)
- Local: data/*.json files after download
b. Data Split & Preprocessor
- Python code/notebook modules that read JSON, produce train/test splits
- Handles “wrong_answers” parsing for multiple-choice extension
c. Evaluation Harness
- lm-evaluation-harness library (external dependency)
- Configuration layer mapping NorMedQA to harness’s Task API
d. LLM Model API
- External service (e.g., OpenAI, Google Gemma) invoked by the harness
e. Metrics & Reporting
- exact_match, rouge1/2/L/Lsum computed by the harness
- Output saved/logged by script or notebook
f. Orchestration
- Jupyter notebook for guided setup & execution
- CLI Python script for automated runs in CI or local shell

##### Relationships & Data Flow
1. User/CI triggers the notebook or python script
2. Script downloads (or reads) JSON dataset from data/
3. Preprocessor component splits and cleans data
4. Harness component reads preprocessed data, invokes LLM API for each question
5. LLM returns answers; harness computes metrics
6. Metrics and raw predictions output to console, log files, or notebook cells

##### Architectural Patterns & Principles
- Modular pipeline: clear separation between Data Ingestion, Processing, Evaluation, Reporting
- Extensibility: “wrong_answers” field enables future multiple-choice testing
- Leverage existing libraries (Don’t reinvent the harness; plug in NorMedQA as a new Task)
- Reproducibility: JSON-based config, notebook walkthrough, script automation

- Clear, consistent labels on every component
- Local vs external
- Data vs compute vs storage
- Title and version annotation (e.g., “NorMedQA v2 Benchmark Architecture”)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Title
    title["NorMedQA v2 Benchmark Architecture"]
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        Zenodo["Zenodo (External Hosting)"]:::external
        FullDataset["norwegian_medical_qa_v2.json"]:::storage
        TextOnly["norwegian_medical_qa_v2_txt_only.json"]:::storage
        README["README.md"]:::output
    end

    %% Processing Layer
    subgraph "Processing Layer"
        direction TB
        Preprocessor["CLI Script: norwegian_medical_qa.py"]:::module
        Notebook["Interactive Notebook: norwegian_medical_qa.ipynb"]:::module
    end

    %% Evaluation Layer
    subgraph "Evaluation Layer"
        direction TB
        Harness["lm-evaluation-harness (External)"]:::external
        LLMAPI["LLM Model API (OpenAI/Gemma)"]:::external
    end

    %% Reporting Layer
    subgraph "Reporting Layer"
        direction TB
        Metrics["Metrics & Logs"]:::output
    end

    %% Connections
    Zenodo -->|"raw JSON"| FullDataset
    Zenodo -->|"raw JSON"| TextOnly
    FullDataset -->|"raw JSON"| Preprocessor
    TextOnly -->|"raw JSON"| Preprocessor
    Preprocessor -->|"train/test splits"| Harness
    Harness -->|"prompt & response"| LLMAPI
    LLMAPI -->|"response"| Harness
    Harness -->|"metrics"| Metrics
    Notebook -->|"trigger pipeline"| Preprocessor
    README --- Zenodo

    %% Click Events
    click FullDataset "https://github.com/kelkalot/normedqa/blob/main/data/norwegian_medical_qa_v2.json"
    click TextOnly "https://github.com/kelkalot/normedqa/blob/main/data/norwegian_medical_qa_v2_txt_only.json"
    click Preprocessor "https://github.com/kelkalot/normedqa/blob/main/code/norwegian_medical_qa.py"
    click Notebook "https://github.com/kelkalot/normedqa/blob/main/code/norwegian_medical_qa.ipynb"
    click README "https://github.com/kelkalot/normedqa/blob/main/README.md"

    %% Styles
    classDef module fill:#cfe2ff,stroke:#4177f6,stroke-width:1px;
    classDef storage fill:#d1e7dd,stroke:#198754,stroke-width:1px,shape:cylinder;
    classDef external fill:#fddfbe,stroke:#f29400,stroke-width:1px,shape:cloud;
    classDef output fill:#e2e3e5,stroke:#6c757d,stroke-width:1px,shape:note;
```
