# docto-rin/igakuqa119 architecture

> Architecture diagram of the GitHub repository docto-rin/igakuqa119, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/docto-rin/igakuqa119
- Repository: https://github.com/docto-rin/igakuqa119

## Overview

The IgakuQA119 project is a specialized evaluation framework to test and benchmark Large Language Models (LLMs) using exam questions from Japan’s Medical Licensing Examination. Its primary purpose is to drive LLMs (both via cloud APIs and local setups using Ollama) through processing exam questions and then consolidate and grade those answers, updating a leaderboard based on performance metrics.

##### Main Components
a. Environment & Configuration:
- Files like .env.example and pyproject.toml indicate that environment variables and project dependencies are critical to run experiments (e.g., setting API keys for cloud-based LLMs or configuring local Ollama instances).
b. Question & Data Input Module:
- All exam questions are stored in JSON format in the “questions” directory.
- Scripts like “txt2json.py” and “extract_question_info.py” in the prepro_utils submodule are used to prepare or transform input data.
c. Core Evaluation Engine:
- main.py acts as the primary script that orchestrates an experiment. It likely calls upon modules in “src” (like llm_solver.py and process_llm_output.py) to perform the actual LLM query/response handling.
- grade_answers.py is used post-processing to evaluate answers and update the leaderboard.
d. Result & Post-processing Unit:
- The “results” directory holds output files (CSV and JSON leaderboard data).
- Scripts like scripts/merge_results.py and rerun_skipped.py handle the reprocessing and merging of answer data for cases with skipped questions.
e. Utilities and Helpers:
- The scripts under “scripts/prepro_utils” (such as cleaning_ocr_txt.py, pdf2txt.py, etc.) indicate additional preprocessing steps to convert raw OCR or PDF data into usable formats.
- This module can be visualized as a supportive pipeline feeding into the main evaluation cycle.

##### Relationships and Interactions
- The system begins with the ingestion of exam questions (from “questions/…” JSON files).
- The main controller (main.py) coordinates tests against one or more LLMs. It dynamically selects between cloud-based LLM APIs or a local instance (like via Ollama).
- The llm_solver and process_llm_output components handle sending requests to the LLMs and parsing their responses.
- After the first run, skipped or incomplete questions are isolated (using rerun_skipped.py) and reprocessed. Their outcomes are then merged (using merge_results.py) with the original set.
- Finally, grade_answers.py processes all the consolidated outputs to compute scores and update the leaderboard stored in the “results” folder and reflected within the README.

##### Architectural Patterns and Design Principles
- The project employs a modular design by separating concerns across different directories: core logic in “src”, utilities under “scripts”, and data inputs/outputs in “questions” and “results.”
- It uses scripting for orchestrating workflows, showing a pipeline-like structure where data flows from preprocessing, evaluation, to grading.
- There is a clear separation between the evaluation engine (LLM interaction) and post-processing/grading modules. This indicates a decoupled and easily extendable design.

##### Relevant Technologies and Frameworks
- Python 3.10+ is required, and dependency management appears to be handled via the “uv” tool and pyproject.toml.
- Cloud API interactions (e.g., OpenAI, Anthropic, Gemini) as well as local LLM access via Ollama reveal a hybrid model integration.
- The use of JSON for questions and CSV/JSON for result outputs indicates standardized data interchange formats.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Environment & Configuration
    subgraph "Environment & Config"
        Env1["Environment Variables (.env.example)"]:::config
        Env2["Project Dependencies (pyproject.toml)"]:::config
    end

    %% Question Loader/Preprocessor
    subgraph "Question Loader/Preprocessor"
        QFolder["Questions Folder"]:::input
        Txt2Json["txt2json (scripts/prepro_utils/txt2json.py)"]:::input
        ExtractQ["extract_question_info (scripts/prepro_utils/extract_question_info.py)"]:::input
    end

    %% Utilities & Helpers
    subgraph "Utilities & Helpers"
        Util1["cleaning_ocr_txt (scripts/prepro_utils/cleaning_ocr_txt.py)"]:::util
        Util2["pdf2txt (scripts/prepro_utils/pdf2txt.py)"]:::util
        Util3["create_correct_answers_csv (scripts/prepro_utils/create_correct_answers_csv.py)"]:::util
        Util4["test_model_parameters (scripts/prepro_utils/test_model_parameters.py)"]:::util
    end

    %% Core Evaluation Engine
    subgraph "Core Evaluation Engine"
        Engine1["Main Orchestrator (main.py)"]:::engine
        LlmSolver["LLM Solver (src/llm_solver.py)"]:::engine
        ProcessOutput["LLM Output Processor (src/process_llm_output.py)"]:::engine
        GradeAnswers["Grade Answers (grade_answers.py)"]:::engine
    end

    %% Result & Post-processing Unit
    subgraph "Result & Post-processing Unit"
        MergeResults["Merge Results (scripts/merge_results.py)"]:::grading
        RerunSkipped["Rerun Skipped (rerun_skipped.py)"]:::grading
        Results["Results Folder"]:::storage
    end

    %% Data Flow Connections
    %% Environment configuration influences the main orchestrator
    Env1 -->|"config"| Engine1
    Env2 -->|"config"| Engine1

    %% Utilities transform raw data into structured questions
    Util1 -->|"transforms"| QFolder
    Util2 -->|"transforms"| QFolder
    Util3 ---|"support"| QFolder
    Util4 ---|"support"| QFolder

    %% Question Loader flows into Core Engine
    QFolder -->|"input questions"| Engine1
    Txt2Json --> QFolder
    ExtractQ --> QFolder

    %% Core Evaluation Engine internal flow
    Engine1 -->|"calls"| LlmSolver
    LlmSolver -->|"processes"| ProcessOutput

    %% Branch to Result Post-processing
    ProcessOutput -->|"initial output"| RerunSkipped
    ProcessOutput -->|"initial output"| MergeResults
    RerunSkipped -->|"refined output"| MergeResults
    MergeResults -->|"consolidated results"| GradeAnswers
    GradeAnswers -->|"updates"| Results

    %% Styles
    classDef config fill:#FFD700,stroke:#000,stroke-width:2px;
    classDef input fill:#ADFF2F,stroke:#000,stroke-width:2px;
    classDef util fill:#87CEEB,stroke:#000,stroke-width:2px;
    classDef engine fill:#FFB6C1,stroke:#000,stroke-width:2px;
    classDef grading fill:#DDA0DD,stroke:#000,stroke-width:2px;
    classDef storage fill:#90EE90,stroke:#000,stroke-width:2px;

    %% Click Events for Environment & Config
    click Env1 "https://github.com/docto-rin/igakuqa119/blob/main/.env.example"
    click Env2 "https://github.com/docto-rin/igakuqa119/blob/main/pyproject.toml"

    %% Click Events for Question & Data Input Module
    click QFolder "https://github.com/docto-rin/igakuqa119/tree/main/questions/"
    click Txt2Json "https://github.com/docto-rin/igakuqa119/blob/main/scripts/prepro_utils/txt2json.py"
    click ExtractQ "https://github.com/docto-rin/igakuqa119/blob/main/scripts/prepro_utils/extract_question_info.py"

    %% Click Events for Core Evaluation Engine
    click Engine1 "https://github.com/docto-rin/igakuqa119/blob/main/main.py"
    click LlmSolver "https://github.com/docto-rin/igakuqa119/blob/main/src/llm_solver.py"
    click ProcessOutput "https://github.com/docto-rin/igakuqa119/blob/main/src/process_llm_output.py"
    click GradeAnswers "https://github.com/docto-rin/igakuqa119/blob/main/grade_answers.py"

    %% Click Events for Result & Post-processing Unit
    click MergeResults "https://github.com/docto-rin/igakuqa119/blob/main/scripts/merge_results.py"
    click RerunSkipped "https://github.com/docto-rin/igakuqa119/blob/main/rerun_skipped.py"
    click Results "https://github.com/docto-rin/igakuqa119/tree/main/results/"

    %% Click Events for Utilities & Helpers
    click Util1 "https://github.com/docto-rin/igakuqa119/blob/main/scripts/prepro_utils/cleaning_ocr_txt.py"
    click Util2 "https://github.com/docto-rin/igakuqa119/blob/main/scripts/prepro_utils/pdf2txt.py"
    click Util3 "https://github.com/docto-rin/igakuqa119/blob/main/scripts/prepro_utils/create_correct_answers_csv.py"
    click Util4 "https://github.com/docto-rin/igakuqa119/blob/main/scripts/prepro_utils/test_model_parameters.py"
```
