How esmaeilnourani/lsfo-expansion works
Project type and purpose
• This is an open-source, research-oriented pipeline for recognizing lifestyle factors (LSFs) in biomedical text.
• It provides two NER systems (dictionary/tagger-based and transformer-based), an ontology (LSFO), model training, large-scale inference over PubMed/PMC, and benchmarking/evaluation.
Main architectural components
A. External services & resources
– PubMed API & PMC Open Access bulk articles
– Zenodo repository for LSFO, trained models, tagger dictionaries
– Git submodules:
• LSFO (Lifestyle Factors Ontology repository)
• S1000_Transformer_NER (Transformer NER training code)
• S1000_Transformer_Tagger (Transformer NER tagging code)
B. Data layer (on‐disk storage under data/ and xref/)
– Raw documents and gold annotations (brat .ann/.txt files)
– Balanced train/test splits, merged TSV files
– Xref ontology spreadsheets and text corpora
– Benchmark outputs and plot input TSVs
– Pickled intermediate results (e.g., distances_candidates_from_lsf_and_non_lsf.pickle)
C. Utilities & preprocessing
– utils/retrive_PubMed.py, utils/retrieve_LSFC.py: fetch and preprocess raw text and LSFC metadata
– utils/semantic_term_placement.py, IAA.py: annotation assistance and inter-annotator agreement
– standoff2conll: conversion between brat standoff and CoNLL format for transformer NER
D. Model training & expansion
– src/LSFC-expansion notebooks: BERTopic + KNN for dictionary expansion
– src/Xrefs notebooks: ontology selection and automatic annotation
– S1000_Transformer_NER submodule:
• scripts/run-ner.sh → fine-tune BioBERT (transformer NER) on annotated corpus
– setup.sh & requirements.txt: environment setup (conda, Python 3.9, spaCy, scispaCy, transformers)
E. Tagging/inference
– S1000_Transformer_Tagger submodule:
• scripts/run-bio-tagger.sh → load trained transformer model + apply to new documents
– Dictionary tagger (in NER-Benchmarking/tagger_input & tagger_output) for rule-based tagging
F. Benchmarking & evaluation
– data/NER-Benchmarking directory: raw annotations, filtered/categorized splits, benchmark results
– src/NER-Benchmarking/scripts: precision‐recall plotting, OOC handling, performance progression
– Jupyter notebooks (1–4) for abstract selection, annotation agreement, tagger vs transformer benchmarking
G. Results & visualization
– plots/ directory for generated figures
– benchmark_results/*.txt and *.tsv for detailed performance metrics
Relationships & data flow
1. External data (PubMed, PMC) and Zenodo files → Data layer
2. utils scripts → preprocess raw text, generate standoff annotations
3. standoff2conll → prepare CoNLL input for transformer NER
4. Training branch:
• src/LSFC-expansion & src/Xrefs notebooks → expand/enrich dictionary → feed into tagger and benchmarking
• S1000_Transformer_NER → fine-tune model → save to disk
5. Inference branch:
• Dictionary tagger reads dictionary files → generate tagger_output
• S1000_Transformer_Tagger reads transformer model + standoff2conll output → generate tagger_output
6. Benchmarking:
• Compare tagger_output vs gold annotations in data/NER-Benchmarking
• src/NER-Benchmarking/scripts compute precision, recall, F1 → write benchmark_results
7. Visualization:
• Notebooks & scripts in src/NER-Benchmarking plot results → plots/
Architectural patterns & design principles
• Modular separation: distinct submodules for training, tagging, benchmarking, expansion, retrieval
• Batch pipeline: data flows through preprocessing → training/inference → evaluation → visualization
• Research notebooks for exploratory analysis, with scripts for productionized steps
• Git submodules to reuse external codebases without duplication
• File-based storage for corpora, annotations, model checkpoints