How jamal-20/genai-hta-consistency works
This proof-of-concept extracts five PICO fields from a NICE appraisal report in five Gemini runs, checks field-level consistency, then simulates review of flagged disagreements and evaluates single-run and reviewed results against a gold standard. The graph follows the documented workflow and sampled source; evaluation’s gold standard is embedded in its script, and the review is simulated rather than an interactive researcher step.
Main entry points: evaluate.py Evaluation and review · extract_multiple.py Repeated extraction · utils.py LLM and data helpers · check_consistency.py Consistency check
Components 16
Extraction Inputs
- NICE report text
data/ta1074_clinical_effectiveness.txt - Extraction prompt
prompts/extraction_prompt.txt
Extraction Pipeline
- Repeated extraction
src/extract_multiple.py - LLM and data helpers
src/utils.py - Five extraction runs
results/multiple_runs.json
Consistency and Evaluation
- Consistency check
src/check_consistency.py - Consistency table
results/consistency_table.csv - Evaluation and review
src/evaluate.py - Simulated review
src/evaluate.py - Embedded gold standard
src/evaluate.py - Comparison table
results/comparison_table.csv - Review detail
results/researcher_in_loop.csv - Error analysis
analysis/error_analysis.md - HTA reflection
analysis/hta_context_reflection.md
Other components
- Project user
- Google Gemini
Connections 19
- Project user → Repeated extraction: starts extraction
- Repeated extraction → NICE report text: reads report
- Repeated extraction → Extraction prompt: reads prompt
- Repeated extraction → LLM and data helpers: calls helpers
- LLM and data helpers → Google Gemini: sends prompt
- LLM and data helpers → Repeated extraction: returns response
- Repeated extraction → Five extraction runs: writes runs
- Project user → Consistency check: starts check
- Consistency check → Five extraction runs: reads runs
- Consistency check → LLM and data helpers: calls matcher
- Consistency check → Consistency table: writes table
- Project user → Evaluation and review: starts evaluation
- Evaluation and review → Five extraction runs: reads runs
- Evaluation and review → Consistency table: reads flags
- Evaluation and review → Simulated review: simulates review
- Evaluation and review → Embedded gold standard: compares against
- Evaluation and review → LLM and data helpers: calls matcher
- Evaluation and review → Comparison table: writes comparison
- Evaluation and review → Review detail: writes review detail