# dianbowork/spn4re architecture

> Architecture diagram of the GitHub repository dianbowork/spn4re, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dianbowork/spn4re
- Repository: https://github.com/dianbowork/spn4re

## Overview

##### Project type and purpose
- This is an open-source NLP research tool that implements and trains a joint entity and relation extraction model using Set Prediction Networks (SPN). It is a machine-learning pipeline, not a web app or microservice suite. The goal is to load benchmark datasets (NYT, WebNLG), preprocess them, instantiate a PyTorch/Transformers-based model, train it, and evaluate performance.

##### File structure
- Top-level:
- main.py — entry point, parses CLI args, kicks off training/evaluation
- data/ — raw datasets and preprocessing scripts for NYT and WebNLG
- models/ — model components: sequence encoder, set decoder, matcher, loss criterion, high-level SetPred4RE module
- trainer/ — Trainer class encapsulating the training loop, optimization, checkpointing, logging
- utils/ — helpers for data loading, metrics, logging (average meters), alphabet handling, generic functions

- Patterns:
- Modular ML pipeline with clear separation: data prep → model definition → trainer → evaluation
- Use of external resources: BERT checkpoint directory, PyTorch, Transformers

##### README for additional insights
- Confirms use of PyTorch ≥1.5.0, Transformers 2.6.0, Python 3.7
- Shows CLI commands that pass BERT directory, training hyperparameters, dataset choices (NYT partial/exact, WebNLG)
- Mentions that trained model parameters are downloadable separately

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Preparation Stage
    subgraph "Data Preparation"
        direction TB
        raw_casrel["Raw NYT CASREL Data"]:::data
        raw_exact["Raw NYT Exact Data"]:::data
        webnlg_raw["WebNLG Raw Data"]:::data
        casrel_nb["Untitled.ipynb"]:::processing
        exact_nb["Deduplication.ipynb"]:::processing
        build_webnlg["build_data.py"]:::processing
        clean_webnlg["Cleaned WebNLG JSONs"]:::data
    end

    %% Dataset & Utils Stage
    subgraph "Dataset & Utilities"
        direction TB
        dataset_loader["Dataset & DataLoader Definitions"]:::processing
        alphabet["Alphabet Handling Module"]:::processing
        avg_meter["Average Meter Utility"]:::processing
        metrics["Metric Computation Utilities"]:::processing
        helpers["Generic Helper Functions"]:::processing
    end

    %% Model Architecture Stage
    subgraph "Model Architecture"
        direction TB
        bert_ext["Pretrained BERT Encoder"]:::external
        seq_enc["Sequence Encoder"]:::model
        set_dec["Set Decoder"]:::model
        matcher["Hungarian Matcher"]:::model
        criterion["Set Prediction Loss Criterion"]:::model
        spn_model["SPN4RE Model Orchestrator"]:::model
    end

    %% Training Pipeline Stage
    subgraph "Training Pipeline"
        direction TB
        cli["CLI Orchestrator"]:::processing
        trainer["Trainer"]:::processing
        checkpoints["Model Checkpoints"]:::data
        logs["Logs"]:::data
        eval_metrics["Evaluation Metrics"]:::data
    end

    %% External Resources
    subgraph "External Resources"
        direction TB
        gpu["GPU/CPU Compute"]:::external
    end

    %% Data Flow
    raw_casrel -->|preprocess| casrel_nb
    raw_exact -->|preprocess| exact_nb
    webnlg_raw -->|preprocess| build_webnlg
    build_webnlg -->|outputs| clean_webnlg
    casrel_nb -->|outputs cleaned JSON| dataset_loader
    exact_nb -->|outputs cleaned JSON| dataset_loader
    clean_webnlg -->|outputs cleaned JSON| dataset_loader

    %% Dataset to Trainer
    dataset_loader --> cli
    alphabet --> cli
    avg_meter --> trainer
    metrics --> trainer
    helpers --> trainer

    %% CLI orchestration
    cli -->|instantiates| trainer
    cli -->|loads model| spn_model

    %% Trainer and Model Interaction
    trainer -->|forward/backward| spn_model
    spn_model -->|loads pretrained| bert_ext
    spn_model --> seq_enc
    spn_model --> set_dec
    set_dec --> matcher
    set_dec --> criterion
    trainer --> checkpoints
    trainer --> logs
    trainer --> eval_metrics
    trainer --> gpu

    %% Click Events
    click cli "https://github.com/dianbowork/spn4re/blob/main/main.py"
    click raw_casrel "https://github.com/dianbowork/spn4re/tree/main/data/NYT/casrel_data/"
    click raw_exact "https://github.com/dianbowork/spn4re/tree/main/data/NYT/exact_data/"
    click casrel_nb "https://github.com/dianbowork/spn4re/blob/main/data/NYT/casrel_data/Untitled.ipynb"
    click exact_nb "https://github.com/dianbowork/spn4re/blob/main/data/NYT/exact_data/Deduplication.ipynb"
    click build_webnlg "https://github.com/dianbowork/spn4re/blob/main/data/WebNLG/build_data.py"
    click clean_webnlg "https://github.com/dianbowork/spn4re/tree/main/data/WebNLG/clean_WebNLG/"
    click dataset_loader "https://github.com/dianbowork/spn4re/blob/main/utils/data.py"
    click alphabet "https://github.com/dianbowork/spn4re/blob/main/utils/alphabet.py"
    click avg_meter "https://github.com/dianbowork/spn4re/blob/main/utils/average_meter.py"
    click metrics "https://github.com/dianbowork/spn4re/blob/main/utils/metric.py"
    click helpers "https://github.com/dianbowork/spn4re/blob/main/utils/functions.py"
    click seq_enc "https://github.com/dianbowork/spn4re/blob/main/models/seq_encoder.py"
    click set_dec "https://github.com/dianbowork/spn4re/blob/main/models/set_decoder.py"
    click matcher "https://github.com/dianbowork/spn4re/blob/main/models/matcher.py"
    click criterion "https://github.com/dianbowork/spn4re/blob/main/models/set_criterion.py"
    click spn_model "https://github.com/dianbowork/spn4re/blob/main/models/setpred4RE.py"
    click trainer "https://github.com/dianbowork/spn4re/blob/main/trainer/trainer.py"

    %% Styles
    classDef data fill:#D0E6F6,stroke:#4A90E2,color:#000;
    classDef processing fill:#E2F0D9,stroke:#6AA84F,color:#000;
    classDef model fill:#FDEBD0,stroke:#E69138,color:#000;
    classDef external fill:#F2F2F2,stroke:#999999,color:#000;
```
