How kabir5296/banglanlptoolkit works
Main Components
• Core Package Layer: Represent the “banglanlptoolkit” package as the central node.
• Sub-module for Normalization (e.g., bnnlpnormalizer.py and bnnlpnormalizerplus.py). Show that one normalizer is a Unicode normalizer (from an external project) and the other is the specialized normalizer imported via “csebuetnlp normalizer.”
• Sub-module for Tokenization (bntokenizers.py).
• Sub-module for Punctuation Generation (banglapunctuation.py) using the Kaggle dataset model (or HuggingFace repository).
• Sub-module for Text Augmentation (banglaaugmentation.py, banglatextmentation.py, seq2seqaug.py, sequenceclassificationaug.py). Within this component, illustrate:
• Token Replacement module – uses a fill-mask approach with the BanglishBERT Generator.
• Back Translation module – involves interaction with bn-en/en-bn models (using BanglaT5).
• Paraphrasing module – leverages a Bangla paraphrase model.
• Utility functions (utils.py) for shared functionalities.
External Dependencies & Services
• A link from the normalizer modules to the “csebuetnlp/normalizer” external package.
• Connections from the augmentation modules to external HuggingFace models (e.g., BanglishBERT Generator, BanglaT5 models, and the punctuation generator model).
• Optionally denote that the package supports both online (on-the-fly, compose-based stacking) and offline augmentation strategies.
Architectural Patterns and Relationships
• Demonstrate a clear separation of concerns:
• That utility functions support these core functionalities by being called from multiple modules.
Final Checklist
• Notes for versioning (e.g., online augmentation techniques introduced in version 1.1.5).