How anand-subu/automated-clinical-coding-llm works
Project Type and Purpose
• This is a research/script-based implementation of an LLM-guided tree-search algorithm for automated ICD coding in clinical notes.
• It processes Spanish medical notes by first translating them to English (using GPT-3.5), then applies a tree search guided by LLM predictions (either via GPT-3.5 or Llama-2 70B Chat) to assign ICD labels, and finally evaluates the prediction performance.
Main Components (Modules)
A. Data Input & Preprocessing:
• Translation Module:
◦ File: translate_files.py
◦ Purpose: Translates Spanish medical notes to English using GPT-3.5.
• Dataset: The input dataset (e.g., the CodiEsp Dataset) is externally provided.
B. Core Processing – Tree Search Algorithm:
• Tree Search Module:
◦ Files: tree_search_icd.py and run_tree_search.py
◦ Purpose: Implements the iterative tree-search algorithm. It starts at the ICD ontology’s root and, with LLM-guided decision making, navigates through the predictive tree to output the relevant ICD codes.
• Prompt Templates & Helpers:
◦ Files: prompt_templates.py and helpers.py
◦ Purpose: Build and manage the prompts and configuration for interacting with the LLM APIs (for both OpenAI’s GPT variants and DeepInfra’s Llama deployment).
C. Output Predictions:
• Predictions Storage:
◦ Directory: predictions/
◦ Structure: Subdirectories (e.g., GPT-3.5 and Llama-70B) each contain model-specific output files (JSON, TSV, README) for predictions.
D. Evaluation Module:
• Evaluation Script:
◦ File: evaluate_performance.py
◦ Purpose: Evaluates the predicted ICD codes against a gold-standard dataset, converting between JSON and TSV as needed and computing macro- and micro-average metrics.
Relationships and Data Flow
• The process starts with raw Spanish clinical notes provided as input.
• The Translation Module (translate_files.py) consumes these notes, uses GPT-3.5 via the LLM API client (configured in helpers.py), and produces English-translated documents.
• The Tree Search Module (run_tree_search.py and tree_search_icd.py) takes the translated texts, and iteratively queries the LLM (using configurations detailed in helpers.py), utilizing templates from prompt_templates.py. This module logs predictions for each LLM (organized in separate directories under predictions/).
• The Evaluation Module (evaluate_performance.py) then reads the JSON prediction files, converts them if necessary, and compares with reference data to compute performance metrics.
Architectural Patterns and Design Principles
• Modular Pipeline: The system is structured as a sequential pipeline where each module (Translation → Tree Search → Evaluation) is a well-defined stage.
• Separation of Concerns: Each script handles a specific task (translation, prediction, evaluation) and communicates via intermediate files (e.g., translated files, predictions in JSON/TSV).
• External Integration: The architecture clearly shows integration points with external LLM APIs and the use of configuration details (found in helpers.py).