# intelligence-csd-auth-gr/instance-based-zsl architecture

> Architecture diagram of the GitHub repository intelligence-csd-auth-gr/instance-based-zsl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/intelligence-csd-auth-gr/instance-based-zsl
- Repository: https://github.com/intelligence-csd-auth-gr/instance-based-zsl

## Overview

##### Project Type and Purpose
- This is a research‐oriented data‐processing and machine‐learning pipeline implementing an online instance‐based zero‐shot learning (IBZSL) algorithm for semi‐automatic MeSH indexing. It is not a web service or full‐stack app, but an open‐source tool/pipeline that (a) ingests biomedical abstracts, (b) computes embeddings and similarity scores, (c) runs several baseline and proposed zero‐shot methods, (d) integrates external MTI predictions, and (e) evaluates performance.

##### High‐Level Architectural Components
1. Data Storage & Inputs
- Source data (raw MeSH abstracts, frequency CSV, test set archive)
- Pre‐computed caches (embeddings, similarity dictionaries, known‐labels, novel‐labels)
- Artificial and supervised predictions folders
- MTI tool output
2. Pre‐processing Pipeline
- obtain_text_embeddings.py → bioBERT embedding service
- calculate_similarities.py → pairwise cosine computations
- compute_label_similarities_per_pair.py → global label‐to‐label similarity maps
- add_noisy_labels.py → noise injection for imperfect oracle
3. Baselines & External Integrations
- On‐the‐fly NN baseline
- NN_baseline_preprocess.py → distance/similarity per instance
- NN_summary.py → aggregation
- NN_bioBERT_evaluate.py → metrics & plots
- MTI integration
- mti_output_manipulation.py → parse MTI raw output
- mti_predictions.pickle → external supervised predictions
4. IBZSL Core Algorithm
- record_label_similaritiy_scores.py → builds per‐instance DataFrame of novel vs. existing label similarities under different oracle assumptions
- compute_weights_per_instance.py → applies weight equation (max similarity)
- weighted_unweighted_approaches.py → rank‐score computation (Eq.3)
- occurence.py + apply_occurence.py → fine‐grained occurrence‐based filtering and final decision assembly
- IBZSL_evaluate.py → computes Coverage and 1‐Error metrics, histograms and statistics
5. Results & Reporting
- Results folder containing pickled final decisions under ideal, imperfect, realistic oracles, and hybrid modes
- Graph outputs, logs (app.log), and performance tables

##### Inter‐Module Relationships and Data Flow
- Raw Source Data → Pre‐processing → Embeddings / Similarities →
- Feeds both Baselines and IBZSL Core →
- MTI predictions are ingested into IBZSL Core as an external “realistic oracle” branch →
- All branches funnel into respective evaluation scripts →
- Final results stored in Results/ and visualized.

##### Architectural Patterns & Design Principles
- Pipeline/Batch Processing: clearly separate stages (embeddings, similarity, ranking, evaluation).
- Modular layering: Data layer, Pre‐processing layer, Algorithmic layer, Evaluation layer.
- Reusability: pre‐computations and pickles act as memoization.
- Extensibility: each stage is a standalone script that can be swapped or extended (e.g., replace bioBERT with another embedder).

##### Essential Details to Capture
- The two major external dependencies: bioBERT embedding service and MTI tool.
- The three oracle modes (ideal, imperfect with noise, realistic using MTI) branching from the core.
- Pre‐computed caches as a “shortcut” layer.
- The sequential nature of the IBZSL algorithm: similarity recording → weight computation → ranking → occurrence filtering → evaluation.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Storage & Inputs
    subgraph "Data Storage & Inputs"
        rawData["Raw Source Data"]:::data
        precomputedCaches["Pre‐computed Caches"]:::data
        artificialPredictions["Artificial Predictions (mode3)"]:::data
        supervisedPredictions["Supervised Predictions"]:::data
        MTIoutput["MTI Tool Output (.out)"]:::data
        MTIpickle["MTI Predictions (.pickle)"]:::data
    end

    %% Pre-processing Pipeline
    subgraph "Pre-processing Pipeline"
        embed["obtain_text_embeddings.py"]:::preprocess
        calcSim["calculate_similarities.py"]:::preprocess
        computeLabelSim["compute_label_similarities_per_pair.py"]:::preprocess
        addNoise["add_noisy_labels.py"]:::preprocess
    end

    %% Baselines & External Integrations
    subgraph "Baselines & External Integrations"
        nnPre["NN_baseline_preprocess.py"]:::baseline
        nnSummary["NN_summary.py"]:::baseline
        nnEval["NN_bioBERT_evaluate.py"]:::baseline
        mtiAdapter["mti_output_manipulation.py"]:::baseline
    end

    %% IBZSL Core Algorithm
    subgraph "IBZSL Core Algorithm"
        bioBERTInterface["bioBERT Interface"]:::core
        recordSim["record_label_similaritiy_scores.py"]:::core
        computeWeights["compute_weights_per_instance.py"]:::core
        rankScore["weighted_unweighted_approaches.py"]:::core
        occur["occurence.py"]:::core
        applyOccur["apply_occurence.py"]:::core
    end

    %% Evaluation & Reporting
    subgraph "Evaluation & Results"
        ibzslEval["IBZSL_evaluate.py"]:::evaluate
        coverageMetric["coverage_custom_metric.py"]:::evaluate
        supervisedEval["evaluate_modes.py"]:::evaluate
        applicationLog["app.log"]:::evaluate
        finalResults["Results/"]:::evaluate
    end

    %% Core BioBERT Service
    bioBERTInterface -->|embeddings| embed
    rawData -->|raw abstracts| embed
    precomputedCaches -->|cached embeddings| embed

    embed -->|.pickle files| calcSim
    calcSim -->|cosine similarity scores| computeLabelSim
    precomputedCaches -->|cached similarities| computeLabelSim

    computeLabelSim -->|label similarity maps| addNoise
    addNoise -->|noisy labels| recordSim

    %% Baselines Flow
    embed -->|embeddings| nnPre
    calcSim -->|similarities| nnPre
    nnPre -->|"instance similarities"| nnSummary
    nnSummary -->|"aggregated scores"| nnEval
    nnEval -->|plots & metrics| finalResults

    %% MTI Integration
    MTIoutput --> mtiAdapter
    mtiAdapter -->|MTI predictions| recordSim
    MTIpickle --> recordSim

    %% IBZSL Core Flow
    recordSim --> computeWeights
    computeWeights --> rankScore
    rankScore --> occur
    occur --> applyOccur
    applyOccur --> ibzslEval
    ibzslEval -->|Coverage & 1-Error| finalResults

    %% Additional Evaluation
    coverageMetric --> finalResults
    supervisedPredictions --> supervisedEval
    supervisedEval --> finalResults

    %% Cache shortcuts
    precomputedCaches -.-> calcSim
    precomputedCaches -.-> recordSim

    %% Click Events
    click rawData "https://github.com/intelligence-csd-auth-gr/instance-based-zsl/tree/master/source data/"
    click precomputedCaches "https://github.com/intelligence-csd-auth-gr/instance-based-zsl/tree/master/pre‐computed files/"
    click artificialPredictions "https://github.com/intelligence-csd-auth-gr/instance-based-zsl/tree/master/artificial predictions (mode3)/"
    click supervisedPredictions "https://github.com/intelligence-csd-auth-gr/instance-based-zsl/tree/master/Supervised predictions/"
    click MTIoutput "https://github.com/intelligence-csd-auth-gr/instance-based-zsl/blob/master/MTI/mti_output.out"
    click MTIpickle "https://github.com/intelligence-csd-auth-gr/instance-based-zsl/blob/master/MTI/mti_predictions.pickle"
    click embed "https://github.com/intelligence-csd-auth-gr/instance-based-zsl/blob/master/Pre-processing stages/obtain_text_embeddings.py"
    click calcSim "https://github.com/intelligence-csd-auth-gr/instance-based-zsl/blob/master/Pre-processing stages/calculate_similarities.py"
    click computeLabelSim "https://github.com/intelligence-csd-auth-gr/instance-based-zsl/blob/master/Pre-processing stages/compute_label_similarities_per_pair.py"
    click addNoise "https://github.com/intelligence-csd-auth-gr/instance-based-zsl/blob/master/Pre-processing stages/add_noisy_labels.py"
    click nnPre "https://github.com/intelligence-csd-auth-gr/instance-based-zsl/blob/master/on-the-fly baseline/NN_baseline_preprocess.py"
    click nnSummary "https://github.com/intelligence-csd-auth-gr/instance-based-zsl/blob/master/on-the-fly baseline/NN_summary.py"
    click nnEval "https://github.com/intelligence-csd-auth-gr/instance-based-zsl/blob/master/on-the-fly baseline/NN_bioBERT_evaluate.py"
    click mtiAdapter "https://github.com/intelligence-csd-auth-gr/instance-based-zsl/blob/master/MTI/mti_output_manipulation.py"
    click recordSim "https://github.com/intelligence-csd-auth-gr/instance-based-zsl/blob/master/IBZSL/record_label_similaritiy_scores.py"
    click computeWeights "https://github.com/intelligence-csd-auth-gr/instance-based-zsl/blob/master/IBZSL/compute_weights_per_instance.py"
    click rankScore "https://github.com/intelligence-csd-auth-gr/instance-based-zsl/blob/master/IBZSL/weighted_unweighted_approaches.py"
    click occur "https://github.com/intelligence-csd-auth-gr/instance-based-zsl/blob/master/IBZSL/occurence.py"
    click applyOccur "https://github.com/intelligence-csd-auth-gr/instance-based-zsl/blob/master/IBZSL/apply_occurence.py"
    click ibzslEval "https://github.com/intelligence-csd-auth-gr/instance-based-zsl/blob/master/IBZSL/IBZSL_evaluate.py"
    click applicationLog "https://github.com/intelligence-csd-auth-gr/instance-based-zsl/blob/master/IBZSL/app.log"
    click finalResults "https://github.com/intelligence-csd-auth-gr/instance-based-zsl/tree/master/Results/"
    click coverageMetric "https://github.com/intelligence-csd-auth-gr/instance-based-zsl/blob/master/coverage metric/coverage_custom_metric.py"
    click supervisedEval "https://github.com/intelligence-csd-auth-gr/instance-based-zsl/blob/master/Supervised predictions/evaluate_modes.py"

    %% Styles
    classDef data fill:#add8e6,stroke:#333,stroke-width:1px
    classDef preprocess fill:#90ee90,stroke:#333,stroke-width:1px
    classDef baseline fill:#f4a460,stroke:#333,stroke-width:1px
    classDef core fill:#ff7f7f,stroke:#333,stroke-width:1px
    classDef evaluate fill:#dda0dd,stroke:#333,stroke-width:1px
```
