# pierrepo/mdner architecture

> Architecture diagram of the GitHub repository pierrepo/mdner, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pierrepo/mdner
- Repository: https://github.com/pierrepo/mdner

## Overview

##### Project type and purpose
- Project type: A full-stack ML/NER pipeline in Python, focused on named-entity recognition (NER) for molecular dynamics (MD) literature.
- Purpose: Fetch raw MD text data, preprocess and annotate it (automatically and manually), optionally paraphrase for data augmentation, train/evaluate an NER model (spaCy or Transformer-based), analyze results, and expose a Streamlit web app for inference.

##### File structure
Top-level directories/files and their roles:
- binder/ – Conda environment definitions for reproducible runs
- scripts/
– generate_annotation.py: fetches and filters texts, emits JSON/text for annotation
– entity_annotator.py: Streamlit app for manual annotation
– mdner.py & mdner_app.py: training script (CLI) and web-app entrypoint (inference)
– other scripts: run_llm_annotation.py, run_qc_analysis.py, run_scoring_analysis.py, generate_plots_stats.py, mdner (batch build)
- annotations/: human-annotated JSON/text files (input to training)
- ai_annotations/: LLM‐generated annotation experiments
- prompt_templates/: few‐shot prompt files for LLM annotation
- results/
– models/: persisted spaCy/Transformer NER models
– outputs/: evaluation data and metrics CSV
- notebooks/: Jupyter notebooks for exploratory data analysis, QC, visualization
- docs/: markdown for annotation schema, improvement guidelines
- assets/: static images (e.g., annotator GIF)
- README.md & pyproject.toml: project description and dependencies

##### README for additional insights
- Workflow stages documented: annotation generation, manual annotation, paraphrasing, model training/evaluation, and web‐app usage.
- External dependencies: spaCy, HuggingFace models (mbart, Pegasus, etc.), CUDA/GPU, Streamlit.

##### Main components
a) Data Sources
– External MD dataset repository (Zenodo/OSF)
b) Preprocessing & Annotation Pipeline
– generate_annotation.py (text selection, threshold filtering)
– LLM paraphrasing service (HuggingFace API)
– Manual annotation service (Streamlit entity_annotator)
– annotations/ folder (JSON & TXT)
c) Data Store
– annotations/ (human) and ai_annotations/ (LLM)
– prompt_templates/
d) Training Pipeline
– mdner.py CLI (spaCy or Transformer)
– GPU/CPU compute environment (binder/app.yml)
– results/models/ (model artifacts)
e) Evaluation & Analysis
– run_scoring_analysis.py & run_qc_analysis.py
– generate_plots_stats.py
– notebooks/ for interactive EDA and stats
– results/outputs/ (eval_data, results.csv)
f) Deployment Layer
– Streamlit app (scripts/mdner_app.py)
– Web UI for live NER inference
– Model loading from results/models/

##### Architectural patterns & design principles
- Modular pipeline: distinct scripts for each stage.
- Data-centric storage: annotations and results persisted in folders.
- Reproducibility via Binder/Conda environments (binder/*).
- Separation of concerns:
- Data generation/annotation (scripts/generate_annotation.py, entity_annotator.py)
- Model training (scripts/mdner.py)
- Evaluation/analysis (scripts/run_*.py + notebooks)
- Deployment (scripts/mdner_app.py + Streamlit)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Services
    Zenodo["Zenodo/OSF (External MD Dataset)"]:::external
    HuggingFace["HuggingFace API (Paraphrasing)"]:::external

    %% Infrastructure
    binder["Binder Env Definitions"]:::infra
    click binder "https://github.com/pierrepo/mdner/blob/main/binder/environment.yml"

    %% Annotation Pipeline
    genAnn["Annotation Generator"]:::compute
    click genAnn "https://github.com/pierrepo/mdner/blob/main/scripts/generate_annotation.py"
    manualAnn["Manual Annotation Service"]:::compute
    click manualAnn "https://github.com/pierrepo/mdner/blob/main/scripts/entity_annotator.py"
    llmAnn["LLM Annotation Pipeline"]:::compute
    click llmAnn "https://github.com/pierrepo/mdner/blob/main/scripts/run_llm_annotation.py"

    %% Data Stores
    promptTpl["Prompt Templates"]:::datastore
    click promptTpl "https://github.com/pierrepo/mdner/tree/main/prompt_templates/"
    humanData["Human-Annotated Data"]:::datastore
    click humanData "https://github.com/pierrepo/mdner/tree/main/annotations/"
    aiData["AI-Generated Annotation Data"]:::datastore
    click aiData "https://github.com/pierrepo/mdner/tree/main/ai_annotations/"

    %% Training
    trainCLI["Training CLI"]:::compute
    click trainCLI "https://github.com/pierrepo/mdner/blob/main/scripts/mdner.py"
    modelStore["Model Artifacts Store"]:::datastore
    click modelStore "https://github.com/pierrepo/mdner/tree/main/results/models/"

    %% Evaluation & Analysis
    subgraph "Evaluation & Analysis"
        evalScore["Scoring Analysis"]:::compute
        click evalScore "https://github.com/pierrepo/mdner/blob/main/scripts/run_scoring_analysis.py"
        evalQC["QC Analysis"]:::compute
        click evalQC "https://github.com/pierrepo/mdner/blob/main/scripts/run_qc_analysis.py"
        genPlots["Generate Plots & Stats"]:::compute
        click genPlots "https://github.com/pierrepo/mdner/blob/main/scripts/generate_plots_stats.py"
    end
    outputs["Evaluation Outputs Store"]:::datastore
    click outputs "https://github.com/pierrepo/mdner/tree/main/results/outputs/"
    notebooks["EDA Notebooks"]:::compute
    click notebooks "https://github.com/pierrepo/mdner/tree/main/notebooks/"

    %% Deployment
    streamlitApp["Streamlit Inference App"]:::deploy
    click streamlitApp "https://github.com/pierrepo/mdner/blob/main/scripts/mdner_app.py"

    %% Project Metadata
    readme["Project Metadata & Dependencies"]:::infra
    click readme "https://github.com/pierrepo/mdner/blob/main/README.md"

    %% Flows
    Zenodo -->|fetch raw text| genAnn
    genAnn -->|emit raw JSON/text| humanData
    humanData --> manualAnn
    genAnn --> promptTpl
    promptTpl --> llmAnn
    llmAnn --> aiData
    humanData --> trainCLI
    aiData --> trainCLI
    trainCLI --> modelStore
    modelStore --> evalScore
    modelStore --> evalQC
    modelStore --> genPlots
    evalScore --> outputs
    evalQC --> outputs
    genPlots --> outputs
    humanData --> notebooks
    aiData --> notebooks
    outputs --> notebooks
    modelStore --> streamlitApp
    streamlitApp -->|serve inference| endUI[User via Web UI]

    binder --> genAnn
    binder --> trainCLI
    binder --> evalScore
    binder --> streamlitApp

    readme --> genAnn

    %% Styles
    classDef datastore fill:#BBDEFB,stroke:#1E88E5,color:#0D47A1;
    classDef compute fill:#C8E6C9,stroke:#388E3C,color:#1B5E20;
    classDef external fill:#FFE0B2,stroke:#FB8C00,color:#E65100;
    classDef deploy fill:#E1BEE7,stroke:#8E24AA,color:#4A148C;
    classDef infra fill:#E0E0E0,stroke:#9E9E9E,color:#424242;
```
