# grammarly/gector architecture

> Architecture diagram of the GitHub repository grammarly/gector, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/grammarly/gector
- Repository: https://github.com/grammarly/gector

## Overview

##### Main Components
- Data Management & Preprocessing:
- Components include the “data” folder (holding vocabularies and other data files) and utility scripts in the “utils” directory (for example, preprocess_data.py and filter_brackets.py).
- This part covers cleaning, formatting, and preparing raw text data into the specialized format expected by the training procedures.

- Model Core – GEC Module:
- The “gector” directory contains the key code for the model: files for tokenization (tokenization.py, tokenizer_indexer.py), data reading (datareader.py), embedding (bert_token_embedder.py), and model architecture itself (gec_model.py, seq2labels_model.py) alongside the training routines (trainer.py).
- This module is built on PyTorch with dependencies on AllenNLP and Hugging Face’s transformers, leveraging multiple pre-trained encoders (BERT, RoBERTa, XLNet, etc.).

- Training Pipeline:
- Represented by the “train.py” script that invokes the training process, passing parameters like cold_steps_count, transformer_model, etc.
- It links the data input (preprocessed by the utilities) with the model code in the “gector” folder.

- Inference Pipeline:
- The “predict.py” script handles the evaluation/inference process.
- It uses the trained model checkpoint(s) alongside vocabulary files and additional parameters like min_error_probability, additional_confidence, etc.

- Documentation & Configuration:
- The “docs” directory (e.g., training_parameters.md) and configuration files (requirements.txt, README.md) serve as the project’s reference and build configuration.

##### Component Relationships and Data Flow
- Input Data (from external datasets) is preprocessed by utilities.
- Preprocessed data is then used by the Training Pipeline (train.py) which calls modules from the “gector” package.
- During training, models make use of pre-trained Transformer encoders (e.g., BERT, RoBERTa, XLNet) that are embedded within the model architecture.
- The trained model (saved as checkpoints) is later used by the Inference Pipeline (predict.py) for applying error correction to new or test inputs.
- The vocabulary and token mappings stored in the “data/output_vocabulary” directory are referenced both during training and inference.
- An additional branch exists for Text Simplification (sharing much of the same data preparation and training interface, with tweaks such as the --normalize flag and additional filtering steps).

##### Final Touches
- That every component (including configuration repositories like ‘docs’ for training parameters) is represented.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Input
    extData("External Datasets"):::data

    %% Data Management & Preprocessing
    subgraph "Data Management & Preprocessing"
        dataFolder("Data Folder"):::data
        vocabulary("Vocabulary (output_vocabulary)"):::data
        utilsPreprocess("Preprocess Data Script"):::utils
        utilsFilter("Filter Brackets Script"):::utils
    end

    %% Model Core – GEC Module
    subgraph "Model Core - GEC Module"
        tokenization("Tokenization"):::gector
        tokenizerIndexer("Tokenizer Indexer"):::gector
        datareader("Data Reader"):::gector
        bertEmbedder("BERT Token Embedder"):::gector
        gecModel("GEC Model"):::gector
        seq2labels("Seq2Labels Model"):::gector
        gectorTrainer("Training Routine"):::gector
    end

    %% Training Pipeline
    subgraph "Training Pipeline"
        trainScript("Train Script"):::train
    end

    %% Inference Pipeline
    subgraph "Inference Pipeline"
        inferArtifacts("Model Checkpoint & Vocabulary"):::inference
        predictScript("Predict Script"):::inference
    end

    %% Documentation & Configuration
    subgraph "Documentation & Configuration"
        docsDir("Docs Directory"):::doc
        readmeFile("README.md"):::doc
        requirementsFile("requirements.txt"):::doc
    end

    %% External Dependencies
    subgraph "External Dependencies"
        pytorch("PyTorch"):::external
        allennlp("AllenNLP"):::external
        transformers("Transformers"):::external
    end

    %% Data Flow Connections
    extData --> utilsPreprocess
    extData --> utilsFilter
    dataFolder --> vocabulary
    utilsPreprocess --> trainScript
    utilsFilter --> trainScript

    %% Model Core Internal Flow
    tokenization --> tokenizerIndexer
    tokenizerIndexer --> datareader
    datareader --> bertEmbedder
    bertEmbedder --> gecModel
    bertEmbedder --> seq2labels
    gecModel --> gectorTrainer
    seq2labels --> gectorTrainer
    trainScript --> gectorTrainer
    gectorTrainer --> inferArtifacts
    inferArtifacts --> predictScript

    %% External Dependencies Connections
    pytorch --> gectorTrainer
    allennlp --> datareader
    transformers --> bertEmbedder

    %% Text Simplification Branch (dashed arrow)
    trainScript -. "TextSimplification" .-> predictScript

    %% Class Definitions for Colors
    classDef data fill:#fde2e4,stroke:#ffb6c1,stroke-width:2px;
    classDef utils fill:#e2f0cb,stroke:#a2c4a6,stroke-width:2px;
    classDef gector fill:#cde7f0,stroke:#90bad7,stroke-width:2px;
    classDef train fill:#fef9c5,stroke:#f5d76e,stroke-width:2px;
    classDef inference fill:#d5f4e6,stroke:#a1e3a1,stroke-width:2px;
    classDef doc fill:#f0e68c,stroke:#dda0dd,stroke-width:2px;
    classDef external fill:#e6e2af,stroke:#b5ad90,stroke-width:2px;

    %% Click Events for Component Mappings
    click dataFolder "https://github.com/grammarly/gector/tree/master/data/"
    click vocabulary "https://github.com/grammarly/gector/tree/master/data/output_vocabulary"
    click utilsPreprocess "https://github.com/grammarly/gector/blob/master/utils/preprocess_data.py"
    click utilsFilter "https://github.com/grammarly/gector/blob/master/utils/filter_brackets.py"
    click tokenization "https://github.com/grammarly/gector/blob/master/gector/tokenization.py"
    click tokenizerIndexer "https://github.com/grammarly/gector/blob/master/gector/tokenizer_indexer.py"
    click datareader "https://github.com/grammarly/gector/blob/master/gector/datareader.py"
    click bertEmbedder "https://github.com/grammarly/gector/blob/master/gector/bert_token_embedder.py"
    click gecModel "https://github.com/grammarly/gector/blob/master/gector/gec_model.py"
    click seq2labels "https://github.com/grammarly/gector/blob/master/gector/seq2labels_model.py"
    click gectorTrainer "https://github.com/grammarly/gector/blob/master/gector/trainer.py"
    click trainScript "https://github.com/grammarly/gector/blob/master/train.py"
    click predictScript "https://github.com/grammarly/gector/blob/master/predict.py"
    click docsDir "https://github.com/grammarly/gector/tree/master/docs/"
    click readmeFile "https://github.com/grammarly/gector/blob/master/README.md"
    click requirementsFile "https://github.com/grammarly/gector/blob/master/requirements.txt"
```
