How docto-rin/igakuqa119 works
The IgakuQA119 project is a specialized evaluation framework to test and benchmark Large Language Models (LLMs) using exam questions from Japan’s Medical Licensing Examination. Its primary purpose is to drive LLMs (both via cloud APIs and local setups using Ollama) through processing exam questions and then consolidate and grade those answers, updating a leaderboard based on performance metrics.
Main Components
a. Environment & Configuration:
• Files like .env.example and pyproject.toml indicate that environment variables and project dependencies are critical to run experiments (e.g., setting API keys for cloud-based LLMs or configuring local Ollama instances).
b. Question & Data Input Module:
• All exam questions are stored in JSON format in the “questions” directory.
• Scripts like “txt2json.py” and “extract_question_info.py” in the prepro_utils submodule are used to prepare or transform input data.
c. Core Evaluation Engine:
• main.py acts as the primary script that orchestrates an experiment. It likely calls upon modules in “src” (like llm_solver.py and process_llm_output.py) to perform the actual LLM query/response handling.
• grade_answers.py is used post-processing to evaluate answers and update the leaderboard.
d. Result & Post-processing Unit:
• The “results” directory holds output files (CSV and JSON leaderboard data).
• Scripts like scripts/merge_results.py and rerun_skipped.py handle the reprocessing and merging of answer data for cases with skipped questions.
e. Utilities and Helpers:
• The scripts under “scripts/prepro_utils” (such as cleaning_ocr_txt.py, pdf2txt.py, etc.) indicate additional preprocessing steps to convert raw OCR or PDF data into usable formats.
• This module can be visualized as a supportive pipeline feeding into the main evaluation cycle.
Relationships and Interactions
• The system begins with the ingestion of exam questions (from “questions/…” JSON files).
• The main controller (main.py) coordinates tests against one or more LLMs. It dynamically selects between cloud-based LLM APIs or a local instance (like via Ollama).
• The llm_solver and process_llm_output components handle sending requests to the LLMs and parsing their responses.
• After the first run, skipped or incomplete questions are isolated (using rerun_skipped.py) and reprocessed. Their outcomes are then merged (using merge_results.py) with the original set.
• Finally, grade_answers.py processes all the consolidated outputs to compute scores and update the leaderboard stored in the “results” folder and reflected within the README.
Architectural Patterns and Design Principles
• The project employs a modular design by separating concerns across different directories: core logic in “src”, utilities under “scripts”, and data inputs/outputs in “questions” and “results.”
• It uses scripting for orchestrating workflows, showing a pipeline-like structure where data flows from preprocessing, evaluation, to grading.
• There is a clear separation between the evaluation engine (LLM interaction) and post-processing/grading modules. This indicates a decoupled and easily extendable design.
Relevant Technologies and Frameworks
• Python 3.10+ is required, and dependency management appears to be handled via the “uv” tool and pyproject.toml.
• Cloud API interactions (e.g., OpenAI, Anthropic, Gemini) as well as local LLM access via Ollama reveal a hybrid model integration.
• The use of JSON for questions and CSV/JSON for result outputs indicates standardized data interchange formats.