# aimbotparce/mds-mud-namedentityrecognition architecture

> Architecture diagram of the GitHub repository aimbotparce/mds-mud-namedentityrecognition, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aimbotparce/mds-mud-namedentityrecognition
- Repository: https://github.com/aimbotparce/mds-mud-namedentityrecognition

## Overview

##### Project Type and Purpose
- This is a domain-specific NLP pipeline for Named Entity Recognition (NER) on drug-related text, implementing two parallel approaches:
- A classical ML pipeline under `ml/` (feature extraction → model training → prediction → evaluation)
- A neural network pipeline under `nn/` (dataset tensorization → model training → prediction → evaluation)
- It operates on XML documents from a Drug-Drug Interaction dataset (split into train, devel, test in `data/`) and produces BIO-tagged entity labels.

##### File Structure Analysis
- Top-level directories:
- `data/` (raw XML data for train, validation, test)
- `ml/` (feature-based pipeline + subfolder `src/models/` for CRF, Naive Bayes, custom models)
- `nn/` (deep learning pipeline: dataset, codemaps, train/predict/evaluate scripts)
- `.github/` (CI workflows and helper scripts)
- Root scripts: `run-ml.sh`, `run-nn.sh` (orchestrate the two pipelines)
- `requirements.txt` (Python dependencies)
- Separation of concerns: data vs. classical-ML vs. NN vs. CI.

##### README Insights
- Explains B-I-O tagging strategy and entity classes (`drug_n`, `drug`, `group`, `brand`).
- Describes the sequential steps in each pipeline: extraction → train → predict → evaluate.
- Mentions GitHub Actions workflow (`.github/workflows/benchmark-on-pr.yaml`) that runs the full pipeline on every PR.

- Preprocessing Modules:
- ML Feature Extractor (`ml/extract-features.py`)
- NN Dataset Builder (`nn/dataset.py`)
- Model Training Engines:
- Classical ML Trainer (`ml/train.py`)
- Neural Net Trainer (`nn/train.py`)
- Prediction Engines:
- ML Predictor (`ml/predict.py`)
- NN Predictor (`nn/predict.py`)
- Evaluation Modules:
- ML Evaluator (`ml/evaluator.py`)
- NN Evaluator (`nn/evaluator.py`)
- Orchestration: Shell scripts (`run-ml.sh`, `run-nn.sh`) & GitHub Actions (`.github/workflows/benchmark-on-pr.yaml`)
- Artifacts/Outputs: Model files, code maps, prediction/tag files, evaluation reports.

1. Data Layer → Preprocessing
2. Preprocessing → Model Training
3. Trained Model → Prediction Engine
4. Prediction + Ground Truth → Evaluation
5. Orchestration Layer triggers each step in sequence
6. CI/CD (GitHub Actions) triggers Orchestration on push/PR and captures logs/stats

c. Architectural Patterns & Principles:
- Pipeline (linear workflows for ML and NN)
- Modular separation of preprocessing, training, prediction, evaluation
- Parallel “experiments” branches pattern: ML vs. NN as interchangeable strategies
- Reproducibility via scripts and CI

d. Key Technologies & Frameworks:
- Python 3.x, `requirements.txt`
- scikit-learn / CRFs for ML (`ml/src/models/CRF.py`, `multinomial_nb.py`)
- Custom ML models (`ml/src/models/custom.py`)
- PyTorch or TensorFlow (implied in `nn/`)
- Bash scripts (`run-ml.sh`, `run-nn.sh`, `.github/scripts/read_stats.bash`)
- GitHub Actions for CI (`.github/workflows/benchmark-on-pr.yaml`)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CI/CD Orchestration
    subgraph "CI/CD"
        CI["GitHub Actions"]:::ci
        RunMLScript["run-ml.sh"]:::ci
        RunNNScript["run-nn.sh"]:::ci
    end
    CI -->|triggers| RunMLScript
    CI -->|triggers| RunNNScript

    %% Data Layer
    subgraph "Data Layer"
        DataTrain["data/train"]:::data
        DataDevel["data/devel"]:::data
        DataTest["data/test"]:::data
    end

    %% ML Pipeline
    subgraph "Classical ML Pipeline"
        MLFeature["ML Feature Extractor"]:::proc
        FeatureSpace["Feature Space Definition"]:::proc
        MLTrain["Classical ML Trainer"]:::train
        CRFModel["CRF Model"]:::train
        NBModel["Naive Bayes Model"]:::train
        CustomModel["Custom ML Model"]:::train
        MLPredict["ML Predictor"]:::proc
        MLPredUtil["ML Predictions Utility"]:::proc
        MLEval["ML Evaluator"]:::eval
    end

    %% NN Pipeline
    subgraph "Neural Network Pipeline"
        NNDataset["NN Dataset Builder"]:::proc
        CodeMaps["NN Code Maps"]:::proc
        NNTrain["Neural Net Trainer"]:::train
        NNPredict["NN Predictor"]:::proc
        NNEval["NN Evaluator"]:::eval
    end

    %% Dependencies & Documentation
    Requirements["requirements.txt"]:::ext
    DocML["resources/NERC-ML.pdf"]:::ext
    DocNN["resources/NERC-NN.pdf"]:::ext

    %% Orchestration to Pipelines
    RunMLScript --> MLFeature
    RunNNScript --> NNDataset

    %% Data to Preprocessing
    DataTrain --> MLFeature
    DataDevel --> MLFeature
    DataTest --> MLFeature

    DataTrain --> NNDataset
    DataDevel --> NNDataset
    DataTest --> NNDataset

    %% ML Flow
    MLFeature --> FeatureSpace
    FeatureSpace --> MLTrain
    MLTrain --> CRFModel
    MLTrain --> NBModel
    MLTrain --> CustomModel
    MLTrain --> MLPredict
    MLPredict --> MLPredUtil
    MLPredUtil --> MLEval
    DataTest --> MLEval

    %% NN Flow
    NNDataset --> CodeMaps
    CodeMaps --> NNTrain
    NNTrain --> NNPredict
    NNPredict --> NNEval
    DataTest --> NNEval

    %% Extras
    Requirements --- CI
    DocML --- MLTrain
    DocNN --- NNTrain

    %% Click Events
    click DataTrain "https://github.com/aimbotparce/mds-mud-namedentityrecognition/tree/main/data/train"
    click DataDevel "https://github.com/aimbotparce/mds-mud-namedentityrecognition/tree/main/data/devel"
    click DataTest "https://github.com/aimbotparce/mds-mud-namedentityrecognition/tree/main/data/test"
    click MLFeature "https://github.com/aimbotparce/mds-mud-namedentityrecognition/blob/main/ml/extract-features.py"
    click NNDataset "https://github.com/aimbotparce/mds-mud-namedentityrecognition/blob/main/nn/dataset.py"
    click MLTrain "https://github.com/aimbotparce/mds-mud-namedentityrecognition/blob/main/ml/train.py"
    click NNTrain "https://github.com/aimbotparce/mds-mud-namedentityrecognition/blob/main/nn/train.py"
    click MLPredict "https://github.com/aimbotparce/mds-mud-namedentityrecognition/blob/main/ml/predict.py"
    click NNPredict "https://github.com/aimbotparce/mds-mud-namedentityrecognition/blob/main/nn/predict.py"
    click MLEval "https://github.com/aimbotparce/mds-mud-namedentityrecognition/blob/main/ml/evaluator.py"
    click NNEval "https://github.com/aimbotparce/mds-mud-namedentityrecognition/blob/main/nn/evaluator.py"
    click CRFModel "https://github.com/aimbotparce/mds-mud-namedentityrecognition/blob/main/ml/src/models/CRF.py"
    click NBModel "https://github.com/aimbotparce/mds-mud-namedentityrecognition/blob/main/ml/src/models/multinomial_nb.py"
    click CustomModel "https://github.com/aimbotparce/mds-mud-namedentityrecognition/blob/main/ml/src/models/custom.py"
    click FeatureSpace "https://github.com/aimbotparce/mds-mud-namedentityrecognition/blob/main/ml/src/feature_space.py"
    click MLPredUtil "https://github.com/aimbotparce/mds-mud-namedentityrecognition/blob/main/ml/src/predictions.py"
    click CodeMaps "https://github.com/aimbotparce/mds-mud-namedentityrecognition/blob/main/nn/codemaps.py"
    click RunMLScript "https://github.com/aimbotparce/mds-mud-namedentityrecognition/blob/main/run-ml.sh"
    click RunNNScript "https://github.com/aimbotparce/mds-mud-namedentityrecognition/blob/main/run-nn.sh"
    click CI "https://github.com/aimbotparce/mds-mud-namedentityrecognition/blob/main/ .github/workflows/benchmark-on-pr.yaml"
    click CIWorkflow "https://github.com/aimbotparce/mds-mud-namedentityrecognition/blob/main/.github/workflows/benchmark-on-pr.yaml"
    click CIHelper "https://github.com/aimbotparce/mds-mud-namedentityrecognition/blob/main/.github/scripts/read_stats.bash"
    click Requirements "https://github.com/aimbotparce/mds-mud-namedentityrecognition/blob/main/requirements.txt"
    click DocML "https://github.com/aimbotparce/mds-mud-namedentityrecognition/blob/main/resources/NERC-ML.pdf"
    click DocNN "https://github.com/aimbotparce/mds-mud-namedentityrecognition/blob/main/resources/NERC-NN.pdf"

    %% Styles
    classDef data fill:#add8e6,stroke:#000
    classDef proc fill:#98fb98,stroke:#000
    classDef train fill:#ffa500,stroke:#000
    classDef eval fill:#dda0dd,stroke:#000
    classDef ci fill:#d3d3d3,stroke:#000
    classDef ext fill:#f0f8ff,stroke:#000
```
