# prabhashj07/nepalikit architecture

> Architecture diagram of the GitHub repository prabhashj07/nepalikit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/prabhashj07/nepalikit
- Repository: https://github.com/prabhashj07/nepalikit

## Overview

##### Project Type and Purpose
- NepaliKit is an open-source Python library that provides natural language processing tools specifically for the Nepali language.
- Its main features include tokenization (with a built-in SentencePiece-based tokenizer), text preprocessing (including cleaning and stopword handling), and various sentence-level operations.
- The README and file tree indicate that it is not a full-stack application; instead, it is a library with multiple processing components that can be integrated into NLP workflows, including support for PyTorch.

##### Analysis of the File Structure
- At the top level, the project has standard configuration and deployment-related files (e.g., README.md, setup.py, requirements.txt, GitHub workflow configurations).
- The core package is named “nepalikit” and contains several sub-packages:
- manage_stopwords – Handles stopword management (loading and removal).
- preprocessing – Contains the TextProcessor and URL/email handling utilities.
- sentence_operation – Offers modules for extracting, normalizing, and segmenting sentences, as well as handling abbreviations and generating sentence statistics.
- tokenization – Provides both a rule-based Tokenizer and a SentencePiece-based tokenization system (complete with training scripts and pre-built SentencePiece models).
- utils – Includes additional helper functions (e.g., merging or splitting text, word counting) and integration support with PyTorch.
- The “test” directory stands alone with tests that map directly to these components, confirming a modular design suitable for unit testing.

##### Key Components
a. Text Preprocessing:
- TextProcessor class (from preprocessing) along with URL/email handling.
- Common cleaning operations such as removing HTML tags, special characters, stopwords, and extra whitespace.

b. Tokenization:
- Rule-based Tokenizer: Illustrate methods for sentence, word, and character tokenization/detokenization.
- SentencePiece Tokenizer: Indicate its role in tokenization and model training, along with included reference to the pre-trained model files.

c. Stopword Management:
- A component for managing stopwords (loading and filtering stopwords).

d. Sentence Operations:
- Modules for segmentation, extraction, normalization, abbreviation handling, and statistics aggregation.

e. Utility Functions:
- A node for utility functions (from utils) that provide text merging, splitting, and word counting.

##### Relationships and Interactions
- That stopword removal is either part of preprocessing or a separate process that interacts with it.
- Both tokenization streams (rule-based and SentencePiece) may operate on the clean text and provide alternative pipelines for downstream tasks.
- That utility functions may be used to support both tokenization and preprocessing tasks as well as integration with PyTorch-based ML pipelines.

##### Architectural Patterns or Design Principles
- Modular Design: The architecture is clearly separated into self-contained, testable components.
- Single Responsibility: Each module (preprocessing, tokenization, stopword management, sentence operations) is designed to handle a specific aspect of text processing.
- Extensibility: The structure supports the potential for adding more NLP features or integration with other NLP pipelines.

##### Including Relevant Technologies/Frameworks
- Python as the core programming language.
- SentencePiece for subword tokenization.
- PyTorch integration (from util functions) for ML task support.
- GitHub workflows for CI/CD and testing.

## Diagram (Mermaid)

```mermaid
flowchart TD
    A("User/Client Application"):::client
    B("Raw Nepali Text Input"):::client
    A --> B

    subgraph "Preprocessing"
        P1("TextProcessor"):::processing
        P2("URL/Email Handling"):::processing
    end
    B --> P1
    B --> P2

    SM("Stopword Management"):::stopword
    P1 --> SM
    P2 --> SM

    subgraph "Tokenization"
        T1("Rule-based Tokenizer"):::tokenization
        subgraph "SentencePiece Tokenizer"
            T2a("SP Tokenizer API"):::tokenization
            T2b("Training Script"):::tokenization
            T2c("Pre-trained Model Files"):::tokenization
        end
    end
    P1 --> T1
    P1 --> T2a

    SO("Sentence Operations"):::sentence
    P1 --> SO
    P2 --> SO

    U("Utilities"):::utility
    P1 --> U
    T1 --> U
    T2a --> U
    SO --> U

    TC("Testing & CI/CD"):::testing
    TC2("Tests"):::testing
    TC3("GitHub Workflows"):::testing
    U --> TC
    TC --> TC2
    TC --> TC3

    %% Click Events
    click P1 "https://github.com/prabhashj07/nepalikit/blob/main/nepalikit/preprocessing/TextProcessor.py"
    click P2 "https://github.com/prabhashj07/nepalikit/blob/main/nepalikit/preprocessing/urls_emails.py"
    click T1 "https://github.com/prabhashj07/nepalikit/blob/main/nepalikit/tokenization/tokenizer.py"
    click T2a "https://github.com/prabhashj07/nepalikit/blob/main/nepalikit/tokenization/sentencepiece_tokenizer.py"
    click T2b "https://github.com/prabhashj07/nepalikit/blob/main/nepalikit/tokenization/sentencepiece/train.py"
    click T2c "https://github.com/prabhashj07/nepalikit/tree/main/nepalikit/tokenization/sentencepiece/model"
    click SM "https://github.com/prabhashj07/nepalikit/blob/main/nepalikit/manage_stopwords/manage_stopwords.py"
    click SO "https://github.com/prabhashj07/nepalikit/tree/main/nepalikit/sentence_operation"
    click U "https://github.com/prabhashj07/nepalikit/blob/main/nepalikit/utils/utils.py"
    click TC2 "https://github.com/prabhashj07/nepalikit/tree/main/test"
    click TC3 "https://github.com/prabhashj07/nepalikit/tree/main/.github/workflows"

    %% Styles
    classDef client fill:#ffcccc,stroke:#990000,stroke-width:2px;
    classDef processing fill:#ccffcc,stroke:#006600,stroke-width:2px;
    classDef tokenization fill:#cce5ff,stroke:#003366,stroke-width:2px;
    classDef stopword fill:#ffffcc,stroke:#999900,stroke-width:2px;
    classDef sentence fill:#e6ccff,stroke:#663399,stroke-width:2px;
    classDef utility fill:#d9d9d9,stroke:#666666,stroke-width:2px;
    classDef testing fill:#f0e68c,stroke:#b8860b,stroke-width:2px;
```
