How mellandd/hpo-contrastive-embeddings works
Purpose and Execution Model
This is a Python research pipeline for fine-tuning biomedical sentence encoders to align literature-derived HPO phenotype text with ontology-derived disease overlap. It is script-driven, not a service or web application. The main operational entry points are phase scripts: mining, pair generation, training/search, then task-specific evaluation.
1. Literature Corpus Generation
phase1/hpo_textminer.py orchestrates HPO term loading, PubMed/PMC retrieval, XML parsing, and sentence consolidation. It produces the literature context corpus consumed by later stages, typically partitioned Parquet under data/.
• phase1/hpo_textminer.py
• utils/pubmed_fetcher.py
• utils/text_extractor.py
Boundary: NCBI Entrez/PMC is an external, rate-limited source; ENTREZ_EMAIL is required and ENTREZ_API_KEY raises throughput. SciSpaCy supplies biomedical sentence tokenization and filtering.
2. Ontology and Neuro-Symbolic Similarity
HPO terms and their disease annotations are loaded through PyHPO; similarity is computed as Relative Best Pair (RBP), reflecting shared disease associations rather than ontology graph distance. Lin similarity is retained as an ablation baseline.
• utils/hpo_loader.py
• utils/hpo_similarity.py
• utils/config.py
This is the core symbolic boundary: ontology/disease relationships become scalar supervision labels for textual examples.
3. Pair-Dataset Construction
Phase 2 transforms term-associated texts into labeled sentence pairs. It combines same-term positives, moderate-RBP hard negatives, and low-similarity random negatives; it can preassign train/validation/test splits. A parallel generator uses HPO definitions instead of mined literature for controlled comparisons.
• phase2/generate_pair_dataset.py
• phase2/generate_definition_pair_dataset.py
• utils/saver.py
Inputs are Phase 1 Parquet or ontology definitions; outputs are pair datasets and split Parquet files under data/phase2/. This is the main data contract between corpus/ontology processing and ML training.
4. Fine-Tuning and Experiment Selection
Sentence-Transformers/Hugging Face encoders are fine-tuned against pair labels using AnglE (with CoSENT supported), partial transformer-layer freezing, AdamW, and optional AMP. Optuna searches YAML-defined hyperparameter ranges using validation correlation; the batch launcher runs selected model/dataset combinations.
• phase3/train_sentence_transformer.py
• phase3/hyperparameter_search.py
• phase3/run_best_trainings.py
Models originate from the Hugging Face Hub (e.g., PubMedBERT, SapBERT, MiniLM, MPNet) and trained checkpoints are written under models/phase3/.
5. Embedding Construction and Intrinsic Evaluation
Evaluation scripts load a base or fine-tuned encoder, embed text/terms, compute cosine similarities, and compare them to held-out symbolic labels or use retrieval ranks. Supporting builders form per-term text corpora and term embeddings for reusable downstream representations.
• phase4/evaluate_sts.py
• phase4/build_term_text_corpus.py
• phase4/build_term_embeddings.py
6. Downstream Clinical and Literature Retrieval
The primary application boundary evaluates encoder quality against external biomedical datasets: GSC+ mention-to-HPO linking, JAX document-to-HPO retrieval, simulated OMIM disease ranking, and GA4GH Phenopacket patient retrieval.
• phase4/evaluate_gsc.py
• phase4/synthetic_patient_omim.py
• phase4/phenopacket_patient_retrieval.py
These workflows require locally provisioned GSC, JAX, OMIM/HPO-derived data, and Phenopacket Store files; they emit JSON/CSV metrics.
7. Comparative Analysis and Reporting
Phase 4 also coordinates component ablations, transfer-task checks for catastrophic forgetting, subontology classification, and embedding visualizations. Reference outputs are tracked in the repository, while large datasets, checkpoints, and generated embeddings are intentionally excluded.
• phase4/run_pubmedbert_ablation.py
• phase4/evaluate_transfer_tasks.py
• phase4/run_subontology_classification.py
Components 23
Corpus mining
- NCBI Entrez / PMC · Rate-limited literature source.
- HPO text miner · Builds literature sentence corpus.
phase1/hpo_textminer.py - PubMed fetcher
utils/pubmed_fetcher.py - XML text extractor · Uses SciSpaCy sentence filtering.
utils/text_extractor.py - Literature context corpus · Partitioned corpus under data/.
Ontology supervision
- PyHPO ontology data · HPO terms and disease annotations.
- HPO loader
utils/hpo_loader.py - RBP similarity · Computes disease-overlap supervision; Lin is an ablation baseline.
utils/hpo_similarity.py
Pair training
- Literature pair generator · Creates positives and RBP-based hard/random negatives.
phase2/generate_pair_dataset.py - Definition pair generator · Creates controlled ontology-definition pairs.
phase2/generate_definition_pair_dataset.py - Labeled pair datasets · Main corpus-to-training data contract under data/phase2/.
- Sentence encoder trainer · Fine-tunes Hugging Face encoders with AnglE or CoSENT.
phase3/train_sentence_transformer.py - Hyperparameter search · Selects configurations by validation correlation.
phase3/hyperparameter_search.py - Best-training launcher · Runs selected model and dataset combinations.
phase3/run_best_trainings.py - Hugging Face Hub · Supplies base biomedical encoders.
- Fine-tuned checkpoints · Written under models/phase3/.
Evaluation and reporting
- Term text corpus builder · Forms reusable per-term text corpora.
phase4/build_term_text_corpus.py - Term embedding builder · Builds reusable term embeddings.
phase4/build_term_embeddings.py - Intrinsic STS evaluation · Compares cosine similarity with held-out symbolic labels and retrieval ranks.
phase4/evaluate_sts.py - Clinical retrieval evaluation · GSC, OMIM, and Phenopacket retrieval metrics.
phase4/evaluate_gsc.py - Ablation and transfer analysis · Runs ablations, transfer checks, and subontology classification.
phase4/run_pubmedbert_ablation.py - Clinical benchmark data · GSC, JAX, OMIM/HPO-derived, and Phenopacket files.
- Metrics and reports
Connections 29
- NCBI Entrez / PMC → PubMed fetcher: literature retrieval
- PubMed fetcher → HPO text miner: articles
- XML text extractor → HPO text miner: parsed sentences
- HPO text miner → Literature context corpus: writes
- PyHPO ontology data → HPO loader: terms and annotations
- HPO loader → RBP similarity: disease associations
- Literature context corpus → Literature pair generator: term-associated text
- RBP similarity → Literature pair generator: RBP labels
- HPO loader → Definition pair generator: definitions
- RBP similarity → Definition pair generator: RBP labels
- Literature pair generator → Labeled pair datasets: writes splits
- Definition pair generator → Labeled pair datasets: writes splits
- Labeled pair datasets → Hyperparameter search: validation data
- Hyperparameter search → Best-training launcher: selected settings
- Best-training launcher → Sentence encoder trainer: training jobs
- Labeled pair datasets → Sentence encoder trainer: labeled pairs
- Hugging Face Hub → Sentence encoder trainer: base encoder
- Sentence encoder trainer → Fine-tuned checkpoints: writes
- Literature context corpus → Term text corpus builder: sentences
- Term text corpus builder → Term embedding builder: per-term texts
- Fine-tuned checkpoints → Term embedding builder: encoder
- Fine-tuned checkpoints → Intrinsic STS evaluation: encoder
- RBP similarity → Intrinsic STS evaluation: held-out labels
- Fine-tuned checkpoints → Clinical retrieval evaluation: encoder
- Clinical benchmark data → Clinical retrieval evaluation: tasks
- Clinical retrieval evaluation → Metrics and reports: JSON/CSV metrics
- Intrinsic STS evaluation → Metrics and reports: intrinsic metrics
- Fine-tuned checkpoints → Ablation and transfer analysis: model comparisons
- Ablation and transfer analysis → Metrics and reports: reports