# kabir5296/banglanlptoolkit architecture

> Architecture diagram of the GitHub repository kabir5296/banglanlptoolkit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kabir5296/banglanlptoolkit
- Repository: https://github.com/kabir5296/banglanlptoolkit

## Overview

##### Main Components
- Core Package Layer: Represent the “banglanlptoolkit” package as the central node.
- Sub-module for Normalization (e.g., bnnlpnormalizer.py and bnnlpnormalizerplus.py). Show that one normalizer is a Unicode normalizer (from an external project) and the other is the specialized normalizer imported via “csebuetnlp normalizer.”
- Sub-module for Tokenization (bntokenizers.py).
- Sub-module for Punctuation Generation (banglapunctuation.py) using the Kaggle dataset model (or HuggingFace repository).
- Sub-module for Text Augmentation (banglaaugmentation.py, banglatextmentation.py, seq2seqaug.py, sequenceclassificationaug.py). Within this component, illustrate:
- Token Replacement module – uses a fill-mask approach with the BanglishBERT Generator.
- Back Translation module – involves interaction with bn-en/en-bn models (using BanglaT5).
- Paraphrasing module – leverages a Bangla paraphrase model.
- Utility functions (utils.py) for shared functionalities.

##### External Dependencies & Services
- A link from the normalizer modules to the “csebuetnlp/normalizer” external package.
- Connections from the augmentation modules to external HuggingFace models (e.g., BanglishBERT Generator, BanglaT5 models, and the punctuation generator model).
- Optionally denote that the package supports both online (on-the-fly, compose-based stacking) and offline augmentation strategies.

##### Architectural Patterns and Relationships
- Demonstrate a clear separation of concerns:
- That utility functions support these core functionalities by being called from multiple modules.

##### Final Checklist
- Notes for versioning (e.g., online augmentation techniques introduced in version 1.1.5).

## Diagram (Mermaid)

```mermaid
flowchart TD
    A["Raw Text Input"]:::internal
    B["Bangla NLP Toolkit"]:::internal
    K["Processed Text Output"]:::internal
    U["Utility Functions"]:::utility
    X["External Normalizer (csebuetnlp)"]:::external
    Y["HuggingFace Models"]:::external

    %% Normalization Subgraph
    subgraph "Normalization Modules"
        C["Bangla Text Normalizer"]:::internal
        D["Bangla Text Normalizer Plus"]:::internal
    end

    %% Augmentation Aggregator and Components
    M["Augmentation Pipeline"]:::internal
    subgraph "Text Augmentation Modules"
        G["General Augmentation"]:::internal
        H["Text Augmentation Variant"]:::internal
        I["Seq2Seq Augmentation"]:::internal
        J["Sequence Classification Augmentation"]:::internal
    end

    %% Other Core Module
    E["Bangla Tokenizer"]:::internal
    F["Punctuation Generator"]:::internal

    %% Data flow
    A --> B
    B --> C
    B --> D
    C --> E
    D --> E
    E --> F
    F --> M
    M --> K

    %% Augmentation module composition (dashed links to illustrate inclusion)
    M -. "includes" .-> G
    M -. "includes" .-> H
    M -. "includes" .-> I
    M -. "includes" .-> J
    G --> M
    H --> M
    I --> M
    J --> M

    %% External Dependencies
    D -->|"depends_on"| X
    F -->|"calls"| Y
    G -->|"uses"| Y
    H -->|"uses"| Y
    I -->|"uses"| Y
    J -->|"uses"| Y

    %% Utility Functions support (dashed lines)
    U --- C
    U --- D
    U --- E
    U --- F
    U --- G
    U --- H
    U --- I
    U --- J

    %% Click Events for internal components
    click B "https://github.com/kabir5296/banglanlptoolkit/tree/main/banglanlptoolkit/"
    click C "https://github.com/kabir5296/banglanlptoolkit/blob/main/banglanlptoolkit/bnnlpnormalizer.py"
    click D "https://github.com/kabir5296/banglanlptoolkit/blob/main/banglanlptoolkit/bnnlpnormalizerplus.py"
    click E "https://github.com/kabir5296/banglanlptoolkit/blob/main/banglanlptoolkit/bntokenizers.py"
    click F "https://github.com/kabir5296/banglanlptoolkit/blob/main/banglanlptoolkit/banglapunctuation.py"
    click G "https://github.com/kabir5296/banglanlptoolkit/blob/main/banglanlptoolkit/banglaaugmentation.py"
    click H "https://github.com/kabir5296/banglanlptoolkit/blob/main/banglanlptoolkit/banglatextmentation.py"
    click I "https://github.com/kabir5296/banglanlptoolkit/blob/main/banglanlptoolkit/seq2seqaug.py"
    click J "https://github.com/kabir5296/banglanlptoolkit/blob/main/banglanlptoolkit/sequenceclassificationaug.py"
    click U "https://github.com/kabir5296/banglanlptoolkit/blob/main/banglanlptoolkit/utils.py"

    %% Styles
    classDef internal fill:#E3F2FD,stroke:#1E88E5,stroke-width:2px;
    classDef external fill:#FFCDD2,stroke:#E53935,stroke-width:2px;
    classDef utility fill:#C8E6C9,stroke:#43A047,stroke-width:2px;
```
