# esmaeilnourani/lsfo-expansion architecture

> Architecture diagram of the GitHub repository esmaeilnourani/lsfo-expansion, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/esmaeilnourani/lsfo-expansion
- Repository: https://github.com/esmaeilnourani/lsfo-expansion

## Overview

##### Project type and purpose
- This is an open-source, research-oriented pipeline for recognizing lifestyle factors (LSFs) in biomedical text.
- It provides two NER systems (dictionary/tagger-based and transformer-based), an ontology (LSFO), model training, large-scale inference over PubMed/PMC, and benchmarking/evaluation.

##### Main architectural components
A. External services & resources
– PubMed API & PMC Open Access bulk articles
– Zenodo repository for LSFO, trained models, tagger dictionaries
– Git submodules:
- LSFO (Lifestyle Factors Ontology repository)
- S1000_Transformer_NER (Transformer NER training code)
- S1000_Transformer_Tagger (Transformer NER tagging code)

B. Data layer (on‐disk storage under data/ and xref/)
– Raw documents and gold annotations (brat .ann/.txt files)
– Balanced train/test splits, merged TSV files
– Xref ontology spreadsheets and text corpora
– Benchmark outputs and plot input TSVs
– Pickled intermediate results (e.g., distances_candidates_from_lsf_and_non_lsf.pickle)

C. Utilities & preprocessing
– utils/retrive_PubMed.py, utils/retrieve_LSFC.py: fetch and preprocess raw text and LSFC metadata
– utils/semantic_term_placement.py, IAA.py: annotation assistance and inter-annotator agreement
– standoff2conll: conversion between brat standoff and CoNLL format for transformer NER

D. Model training & expansion
– src/LSFC-expansion notebooks: BERTopic + KNN for dictionary expansion
– src/Xrefs notebooks: ontology selection and automatic annotation
– S1000_Transformer_NER submodule:
- scripts/run-ner.sh → fine-tune BioBERT (transformer NER) on annotated corpus
– setup.sh & requirements.txt: environment setup (conda, Python 3.9, spaCy, scispaCy, transformers)

E. Tagging/inference
– S1000_Transformer_Tagger submodule:
- scripts/run-bio-tagger.sh → load trained transformer model + apply to new documents
– Dictionary tagger (in NER-Benchmarking/tagger_input & tagger_output) for rule-based tagging

F. Benchmarking & evaluation
– data/NER-Benchmarking directory: raw annotations, filtered/categorized splits, benchmark results
– src/NER-Benchmarking/scripts: precision‐recall plotting, OOC handling, performance progression
– Jupyter notebooks (1–4) for abstract selection, annotation agreement, tagger vs transformer benchmarking

G. Results & visualization
– plots/ directory for generated figures
– benchmark_results/*.txt and *.tsv for detailed performance metrics

##### Relationships & data flow
1. External data (PubMed, PMC) and Zenodo files → Data layer
2. utils scripts → preprocess raw text, generate standoff annotations
3. standoff2conll → prepare CoNLL input for transformer NER
4. Training branch:
- src/LSFC-expansion & src/Xrefs notebooks → expand/enrich dictionary → feed into tagger and benchmarking
- S1000_Transformer_NER → fine-tune model → save to disk
5. Inference branch:
- Dictionary tagger reads dictionary files → generate tagger_output
- S1000_Transformer_Tagger reads transformer model + standoff2conll output → generate tagger_output
6. Benchmarking:
- Compare tagger_output vs gold annotations in data/NER-Benchmarking
- src/NER-Benchmarking/scripts compute precision, recall, F1 → write benchmark_results
7. Visualization:
- Notebooks & scripts in src/NER-Benchmarking plot results → plots/

##### Architectural patterns & design principles
- Modular separation: distinct submodules for training, tagging, benchmarking, expansion, retrieval
- Batch pipeline: data flows through preprocessing → training/inference → evaluation → visualization
- Research notebooks for exploratory analysis, with scripts for productionized steps
- Git submodules to reuse external codebases without duplication
- File-based storage for corpora, annotations, model checkpoints

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Services & Resources
    subgraph "External Services" 
        direction TB
        PubMed_API["PubMed API"]:::external
        PMC_OA["PMC Open Access Bulk"]:::external
        Zenodo["Zenodo (LSFO, models, dicts)"]:::external
        LSFO_Ontology["LSFO Ontology"]:::external
        Transformer_NER_Training["Transformer NER Training"]:::external
        Transformer_NER_Tagger["Transformer NER Tagging"]:::external
    end

    %% Data Layer
    subgraph "Data Layer" 
        direction TB
        Data_Layer["Raw/Processed Corpora & Annotations"]:::data
        Xref_Ontology["Ontology Xrefs & Spreadsheets"]:::data
        Dictionary_Tagger_Input["Dictionary Tagger Input"]:::data
        Dictionary_Tagger_Output["Dictionary Tagger Output"]:::data
        Benchmark_Results["Benchmark Results"]:::data
    end

    %% Utilities & Preprocessing
    subgraph "Utilities & Preprocessing"
        direction TB
        Preprocessing_Utilities["Retrieval & Annotation Utils"]:::util
        Standoff2CoNLL["Standoff2CoNLL Conversion"]:::util
    end

    %% Model Training & Expansion
    subgraph "Model Training & Expansion"
        direction TB
        Dict_Expansion["Dictionary Expansion Notebooks"]:::model
        Ontology_XRefs["Ontology XRefs & Auto‐Annotation"]:::model
    end

    %% Inference
    subgraph "Inference Engines"
        direction TB
        Dict_Tagger["Dictionary Tagger (rule‐based)"]:::model
        Transformer_Tagger["Transformer Tagger"]:::model
    end

    %% Benchmarking & Evaluation
    subgraph "Benchmarking & Evaluation"
        direction TB
        NER_Benchmark_Notebooks["Benchmarking Notebooks"]:::bench
        Benchmarking_Scripts["Benchmarking & Plotting Scripts"]:::bench
        Tagger_Support_Scripts["Tagger Support Scripts"]:::bench
    end

    %% Visualization
    subgraph "Results & Visualization"
        direction TB
        Visualization_Output["Plots & Figures"]:::bench
    end

    %% Environment
    Environment["Environment Setup (setup.sh, requirements.txt)"]:::env

    %% Data Flows
    PubMed_API --> Data_Layer
    PMC_OA --> Data_Layer
    Zenodo --> Data_Layer

    Data_Layer --> Preprocessing_Utilities
    Xref_Ontology --> Preprocessing_Utilities

    Preprocessing_Utilities --> Standoff2CoNLL
    Standoff2CoNLL --> Transformer_Tagger
    Standoff2CoNLL --> Dict_Tagger

    Dict_Expansion --> Dict_Tagger
    Ontology_XRefs --> Dict_Tagger

    Transformer_NER_Training --> Transformer_Tagger
    Transformer_NER_Training --> Transformer_Tagger

    Dict_Tagger --> NER_Benchmark_Notebooks
    Transformer_Tagger --> NER_Benchmark_Notebooks

    NER_Benchmark_Notebooks --> Benchmarking_Scripts
    Benchmarking_Scripts --> Visualization_Output

    Environment --> Transformer_NER_Training
    Environment --> Transformer_NER_Tagger
    Environment --> NER_Benchmark_Notebooks
    Environment --> Preprocessing_Utilities

    %% Click Events
    click LSFO_Ontology "https://github.com/esmaeilnourani/lsfo-expansion/tree/main/LSFC/"
    click Transformer_NER_Training "https://github.com/esmaeilnourani/lsfo-expansion/tree/main/S1000_Transformer_NER/"
    click Transformer_NER_Tagger "https://github.com/esmaeilnourani/lsfo-expansion/tree/main/S1000_Transformer_Tagger/"
    click Data_Layer "https://github.com/esmaeilnourani/lsfo-expansion/tree/main/data/"
    click Xref_Ontology "https://github.com/esmaeilnourani/lsfo-expansion/tree/main/data/xref/"
    click Dictionary_Tagger_Input "https://github.com/esmaeilnourani/lsfo-expansion/tree/main/data/NER-Benchmarking/tagger/tagger_input/"
    click Dictionary_Tagger_Output "https://github.com/esmaeilnourani/lsfo-expansion/tree/main/data/NER-Benchmarking/tagger/tagger_output/"
    click Benchmark_Results "https://github.com/esmaeilnourani/lsfo-expansion/tree/main/data/NER-Benchmarking/benchmark_results/"
    click Preprocessing_Utilities "https://github.com/esmaeilnourani/lsfo-expansion/tree/main/utils/"
    click Standoff2CoNLL "https://github.com/esmaeilnourani/lsfo-expansion/tree/main/standoff2conll/"
    click Dict_Expansion "https://github.com/esmaeilnourani/lsfo-expansion/tree/main/src/LSFC-expansion/"
    click Ontology_XRefs "https://github.com/esmaeilnourani/lsfo-expansion/tree/main/src/Xrefs/"
    click NER_Benchmark_Notebooks "https://github.com/esmaeilnourani/lsfo-expansion/tree/main/src/NER-Benchmarking/"
    click Benchmarking_Scripts "https://github.com/esmaeilnourani/lsfo-expansion/tree/main/src/NER-Benchmarking/scripts/"
    click Tagger_Support_Scripts "https://github.com/esmaeilnourani/lsfo-expansion/tree/main/src/NER-Benchmarking/scripts/tagger_scripts/"
    click Visualization_Output "https://github.com/esmaeilnourani/lsfo-expansion/tree/main/plots/"
    click Environment "https://github.com/esmaeilnourani/lsfo-expansion/blob/main/setup.sh"

    %% Styles
    classDef external fill:#e3f2fd,stroke-dasharray: 5 5
    classDef data fill:#bbdefb
    classDef util fill:#c8e6c9
    classDef model fill:#ffe0b2
    classDef bench fill:#e1bee7
    classDef env fill:#f5f5f5,stroke:#9e9e9e,stroke-width:1px
```
