How ibtraore/mie-confidence-estimation works
This repository is a research study codebase for clinical information extraction and LLM confidence evaluation. Its user workflow is to provide task-specific annotated CSV data and model configuration, run one of four task scripts, obtain structured model predictions with confidence estimates, and evaluate correctness and uncertainty. The task scripts and shared confidence utilities are excerpted; the broader evaluation metrics are documented in the README, but their full implementation is not shown. The README also notes that the clinical dataset and processing code are not included.
Main entry points: utils.py Confidence utilities · clinical_sense_disambiguation.py Acronym disambiguation · coreference_resolution.py Coreference resolution · medication_attribute_extraction.py Medication attributes
Components 12
Clinical extraction tasks
- Acronym disambiguation
src/clinical_sense_disambiguation.py - Coreference resolution
src/coreference_resolution.py - Medication attributes
src/medication_attribute_extraction.py - Medication status
src/medication_status_extraction.py
Shared runtime
- Model configuration
src/config.py - Confidence utilities
src/utils.py
Confidence evaluation
- Task correctness
src/clinical_sense_disambiguation.py - Confidence metrics
README.md
Other components
- Researcher
- Clinical task data
- Configured LLM
- Prediction and evaluation files
Connections 28
- Researcher → Clinical task data: provides data
README.md - Clinical task data → Acronym disambiguation: supplies examples
README.md - Clinical task data → Coreference resolution: supplies examples
README.md - Clinical task data → Medication attributes: supplies examples
README.md - Clinical task data → Medication status: supplies examples
README.md - Researcher → Model configuration: sets model options
README.md - Acronym disambiguation → Model configuration: reads settings
src/clinical_sense_disambiguation.py - Coreference resolution → Model configuration: reads settings
src/coreference_resolution.py - Medication attributes → Model configuration: reads settings
src/medication_attribute_extraction.py - Medication status → Model configuration: reads settings
src/medication_status_extraction.py - Acronym disambiguation → Configured LLM: invokes model
src/clinical_sense_disambiguation.py - Coreference resolution → Configured LLM: invokes model
src/coreference_resolution.py - Medication attributes → Configured LLM: invokes model
src/medication_attribute_extraction.py - Medication status → Configured LLM: invokes model
src/medication_status_extraction.py - Acronym disambiguation → Confidence utilities: scores token uncertainty
src/clinical_sense_disambiguation.py - Coreference resolution → Confidence utilities: scores token uncertainty
src/coreference_resolution.py - Medication attributes → Confidence utilities: scores token uncertainty
src/medication_attribute_extraction.py - Medication status → Confidence utilities: scores token uncertainty
src/medication_status_extraction.py - Acronym disambiguation → Task correctness: computes correctness
src/clinical_sense_disambiguation.py - Coreference resolution → Confidence utilities: checks answer F1
src/coreference_resolution.py - Medication attributes → Confidence utilities: scores answer matches
src/medication_attribute_extraction.py - Medication status → Confidence utilities: checks answer F1
src/medication_status_extraction.py - Confidence utilities → Confidence metrics: supports confidence scores
README.md - Task correctness → Confidence metrics: evaluates calibration
- Acronym disambiguation → Prediction and evaluation files: writes predictions
src/clinical_sense_disambiguation.py - Coreference resolution → Prediction and evaluation files: writes results
src/coreference_resolution.py - Medication attributes → Prediction and evaluation files: writes results
README.md - Medication status → Prediction and evaluation files: writes results
README.md