# nikitakit/self-attentive-parser architecture

> Architecture diagram of the GitHub repository nikitakit/self-attentive-parser, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nikitakit/self-attentive-parser
- Repository: https://github.com/nikitakit/self-attentive-parser

## Overview

##### Main Components

a) Core Parser Components:
- Parser Interface Layer
- spaCy Integration
- NLTK Integration
- Neural Network Architecture
- Self-Attentive Encoder
- Pre-trained Model Integration (T5, BERT, XLM-R, ELECTRA)
- Parsing Engine
- Chart Parser
- Decoder

b) Data Processing Pipeline:
- Input Processing
- Tokenization
- Sentence Segmentation
- Output Generation
- Parse Tree Construction
- Constituent Labeling

c) Training Infrastructure:
- Model Training Pipeline
- Evaluation System
- Model Export System

##### Relationships and Data Flow

**Raw Text Input -> Tokenization/Segmentation -> Neural Processing -> Parse Tree Output**

**Training Data -> Data Processing -> Model Training -> Model Export -> Deployment**

##### Specific Elements

a) Input Layer:
- Raw text input handlers
- Pre-tokenized input handlers
- Integration points (spaCy/NLTK)

b) Processing Layer:
- Pre-trained model integration
- Self-attentive encoder
- Chart parsing system
- Constituent labeling system

c) Output Layer:
- Parse tree generation
- API outputs (for both spaCy and NLTK)

d) Training Components:
- Training data pipeline
- Model checkpoint system
- Evaluation system (EVALB integration)

##### Important Architectural Notes

- Modular design that allows for multiple pre-trained models
- Parallel processing capabilities
- Language-specific components
- Integration points with external libraries

##### Technical Details

- Support for 11 languages
- GPU/CPU processing options
- Model versioning system
- Integration with HuggingFace models
- Evaluation metrics processing

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Input
        Raw[Raw Text Input]
        PreTok[Pre-tokenized Input]
        Raw & PreTok --> InputHandler[Input Handler]
    end

    subgraph "Parser Interface"
        InputHandler --> SpaCy["spaCy Integration"]:::integration
        InputHandler --> NLTK["NLTK Integration"]:::integration
        click SpaCy "https://github.com/nikitakit/self-attentive-parser/blob/master/src/benepar/integrations/spacy_plugin.py"
        click NLTK "https://github.com/nikitakit/self-attentive-parser/blob/master/src/benepar/integrations/nltk_plugin.py"
    end

    subgraph "Data Processing"
        SpaCy & NLTK --> Tokenizer["Tokenization"]:::process
        Tokenizer --> SentSeg["Sentence Segmentation"]:::process
        click Tokenizer "https://github.com/nikitakit/self-attentive-parser/blob/master/src/benepar/retokenization.py"
    end

    subgraph "Neural Network Architecture"
        SentSeg --> Encoder["Self-Attentive Encoder"]:::core
        Encoder --> CharProc["Character-level Processing"]:::core
        click Encoder "https://github.com/nikitakit/self-attentive-parser/blob/master/src/benepar/partitioned_transformer.py"
        click CharProc "https://github.com/nikitakit/self-attentive-parser/blob/master/src/benepar/char_lstm.py"
    end

    subgraph "Parsing Engine"
        CharProc --> BaseParser["Base Parser"]:::core
        BaseParser --> ChartParser["Chart Parser"]:::core
        ChartParser --> Decoder["Chart Decoder"]:::core
        click BaseParser "https://github.com/nikitakit/self-attentive-parser/blob/master/src/benepar/parse_base.py"
        click ChartParser "https://github.com/nikitakit/self-attentive-parser/blob/master/src/benepar/parse_chart.py"
        click Decoder "https://github.com/nikitakit/self-attentive-parser/blob/master/src/benepar/decode_chart.py"
    end

    subgraph "Training Infrastructure"
        TrainData[(Training Data)]:::data
        TrainData --> DataProc["Data Processing"]:::process
        DataProc --> Training["Model Training"]:::core
        Training --> Eval["Evaluation System"]:::process
        Training --> Export["Model Export"]:::process
        click Training "https://github.com/nikitakit/self-attentive-parser/blob/master/src/main.py"
        click Eval "https://github.com/nikitakit/self-attentive-parser/blob/master/src/evaluate.py"
        click Export "https://github.com/nikitakit/self-attentive-parser/blob/master/src/export.py"
    end

    subgraph "Evaluation Tools"
        Eval --> EVALB["EVALB Scoring"]:::eval
        Eval --> SPMRL["SPMRL Evaluation"]:::eval
        click EVALB "https://github.com/nikitakit/self-attentive-parser/tree/master/EVALB/"
        click SPMRL "https://github.com/nikitakit/self-attentive-parser/tree/master/EVALB_SPMRL/"
    end

    subgraph "Data Management"
        WSJ[(WSJ Dataset)]:::data
        CTB[(CTB Dataset)]:::data
        SPMRL_Data[(SPMRL Dataset)]:::data
        click WSJ "https://github.com/nikitakit/self-attentive-parser/tree/master/data/wsj/"
        click CTB "https://github.com/nikitakit/self-attentive-parser/tree/master/data/ctb_5.1/"
        click SPMRL_Data "https://github.com/nikitakit/self-attentive-parser/tree/master/data/spmrl/"
    end

    Decoder --> ParseTree["Parse Tree Output"]

    %% Styling
    classDef core fill:#2374ab
    classDef process fill:#48a9a6
    classDef integration fill:#4b644a
    classDef data fill:#d4b483,stroke:#c1666b
    classDef eval fill:#c1666b

    %% Legend
    subgraph Legend
        L1[Core Components]:::core
        L2[Processing Steps]:::process
        L3[Integrations]:::integration
        L4[(Data Storage)]:::data
        L5[Evaluation Tools]:::eval
    end
```
