# geekywizkid/jionlp architecture

> Architecture diagram of the GitHub repository geekywizkid/jionlp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/geekywizkid/jionlp
- Repository: https://github.com/geekywizkid/jionlp

## Overview

##### Project type and purpose
- The README emphasizes that JioNLP is a Python library designed for Chinese NLP tasks including preprocessing, parsing, and various helper utilities.
- The project provides several modules covering NLP algorithms, dictionary loading, tools (“gadgets”), rules, text augmentation, and utility functions.
- It is aimed primarily at NLP developers who need ready-made functions to process Chinese text.

##### File structure to outline main components
- At the top level, you have files for licensing, installation (setup.py, requirements.txt), testing (the test/ directory), and documentation.
- The core of the library exists in the “jionlp” directory. Inside, there are clear subdirectories for:
- algorithm: Contains submodules for tasks like BPE encoding, Chinese word segmentation (cws), keyphrase extraction, LLM evaluation (mellm), NER, new word discovery, part-of-speech tagging, sentiment analysis, simhash, summary extraction, and text classification.
- dictionary: Provides various language resources (idioms, locations, stopwords, etc.) and JSON-based dictionaries for LLM tests.
- gadget: A collection of “utility” functions for parsing specific content (e.g., id card, time, phone location, pinyin conversion).
- rule: Contains several rule-based functions for text cleansing and extraction.
- textaug: Provides text augmentation functions such as back translation, homophone substitution, random add/delete, replacement of entities, and character swap functionality.
- util: Contains utility functions for file I/O, logging, timing, and helper annotations.
- The “test” directory provides unit tests that cover many of these functionalities.

##### README for insights
- There are examples of how to load datasets (like the LLM test datasets) and run specific functions, indicating interdependencies (e.g., MELLM in the algorithm package relies on JSON configuration files and points to test scripts for downloading necessary data).

a. Main Components:
– Front-end/User Interface (conceptually): This is the “import jionlp as jio” entry point, where the user’s script acts as the client invoking various functionalities.
– Dictionaries & Data Resources: Represent the “dictionary” component (loading idioms, Chinese texts, LLM test dictionaries, etc.) that feeds data into the algorithms.
– NLP Gadgets: Capture the “gadget” module, which provides functions for parsing id cards, locations, dates, etc.
– Rule-based and Regular Expression Modules: Represent the “rule” package and parts of “algorithm” that clean or transform text.
– Text Augmentation Suite: Represent the “textaug” package, which handles back translation and other data augmentation functionalities.
– Utility Services: Represent the “util” module, which supports all parts with logging, file I/O, and helper functions.

b. Relationships and Interactions:
– Show that NLP Algorithms often obtain their supporting resources from the Dictionaries component. For instance, keyphrase extraction and segmentation might require dictionary lookups.
– Indicate that the Gadget utilities may call algorithm functions or use dictionaries (e.g., parsing a phone number might leverage a dictionary of area codes or invoke a rule-based parser).

c. Architectural Patterns and Design Principles:
– Emphasize modularity: Each sub-package (algorithm, dictionary, gadget, rule, textaug, util) functions as an independent module that interacts via well-defined interfaces.
– Future extensibility: The design anticipates adding more NLP functionalities, so indicate “extensibility points” (e.g., plugin-like modules for additional NLP tasks).

d. Relevant technologies, frameworks, or libraries:
– Python (version >=3.6) and its standard modules.
– Mention potential external dependencies as indicated in requirements.txt.
– Note the integration of MELLM for LLM evaluations, which is a specific algorithm module.

##### Specific elements
- Each component should have a label (e.g., “Algorithm – NER”, “Dictionary Loader”, “Gadget – ID Card Parser”).
- How input text flows into cleansing routines,
- Then to specific algorithm modules,
- And how dictionaries (loaded via “dictionary”) are referenced.
- Adding annotations on critical components, for example:
– “MELLM (LLM Evaluation)” within the algorithm package,
– “Back Translation API” inside text augmentation (indicating external cloud translation services).

##### Putting It All Together
– Branch out to the main sub-components (algorithm, dictionary, gadget, rule, textaug, util).
– Finally, include a “Test/Integration” component linked to the main modules to show how the system’s functionality is verified.

## Diagram (Mermaid)

```mermaid
flowchart TD
    U["User (Client)"]:::user
    J["JioNLP Library"]:::library
    U --> J

    %% Main Components
    J --> A["Core NLP Algorithms"]:::algContainer
    J --> D["Dictionaries & Data Resources"]:::dict
    J --> G["NLP Gadgets"]:::gadget
    J --> R["Rule-based Modules"]:::rule
    J --> T["Text Augmentation Suite"]:::textaug
    J --> Uti["Utility Services"]:::util
    J --> Test["Testing & Integration"]:::test

    %% Core NLP Algorithms Submodules
    subgraph "Core NLP Algorithms Modules"
        direction TB
        BPE["BPE Encoding"]:::alg
        CWS["Chinese Word Segmentation (CWS)"]:::alg
        KP["Keyphrase Extraction"]:::alg
        MELLM["LLM Evaluation (MELLM)"]:::alg
        NER["Named Entity Recognition (NER)"]:::alg
        NW["New Word Discovery"]:::alg
        POS["Part-of-Speech Tagging (POS)"]:::alg
        SENT["Sentiment Analysis"]:::alg
        SIM["Simhash (Similarity Assessment)"]:::alg
        SUM["Summary Extraction"]:::alg
        TC["Text Classification"]:::alg
    end

    %% Relationships Between Components
    D -->|"feeds"| A
    R -->|"cleans"| A
    G -->|"assists"| A
    Uti -->|"supports"| A

    T -->|"augments"| A
    G -->|"lookup"| D

    Test ---|"validates"| A
    Test ---|"validates"| D
    Test ---|"validates"| G
    Test ---|"validates"| R
    Test ---|"validates"| T
    Test ---|"validates"| Uti

    %% Styling Classes Definition
    classDef user fill:#ffcc00,stroke:#333,stroke-width:2px,color:#000;
    classDef library fill:#f0f0f0,stroke:#333,stroke-width:2px,color:#000;
    classDef algContainer fill:#ccffcc,stroke:#333,stroke-width:2px,color:#000;
    classDef alg fill:#b3e6b3,stroke:#333,stroke-width:2px,color:#000;
    classDef dict fill:#cce5ff,stroke:#333,stroke-width:2px,color:#000;
    classDef gadget fill:#ffcccc,stroke:#333,stroke-width:2px,color:#000;
    classDef rule fill:#dddddd,stroke:#333,stroke-width:2px,color:#000;
    classDef textaug fill:#e6ccff,stroke:#333,stroke-width:2px,color:#000;
    classDef util fill:#ffffcc,stroke:#333,stroke-width:2px,color:#000;
    classDef test fill:#d9ead3,stroke:#333,stroke-width:2px,color:#000;

    %% Click Events for Component Mapping
    click A "https://github.com/geekywizkid/jionlp/tree/master/jionlp/algorithm"
    click BPE "https://github.com/geekywizkid/jionlp/tree/master/jionlp/algorithm/bpe"
    click CWS "https://github.com/geekywizkid/jionlp/tree/master/jionlp/algorithm/cws"
    click KP "https://github.com/geekywizkid/jionlp/tree/master/jionlp/algorithm/keyphrase"
    click MELLM "https://github.com/geekywizkid/jionlp/tree/master/jionlp/algorithm/mellm"
    click NER "https://github.com/geekywizkid/jionlp/tree/master/jionlp/algorithm/ner"
    click NW "https://github.com/geekywizkid/jionlp/tree/master/jionlp/algorithm/new_word"
    click POS "https://github.com/geekywizkid/jionlp/tree/master/jionlp/algorithm/pos"
    click SENT "https://github.com/geekywizkid/jionlp/tree/master/jionlp/algorithm/sentiment"
    click SIM "https://github.com/geekywizkid/jionlp/tree/master/jionlp/algorithm/simhash"
    click SUM "https://github.com/geekywizkid/jionlp/tree/master/jionlp/algorithm/summary"
    click TC "https://github.com/geekywizkid/jionlp/tree/master/jionlp/algorithm/text_classification"
    click D "https://github.com/geekywizkid/jionlp/tree/master/jionlp/dictionary"
    click G "https://github.com/geekywizkid/jionlp/tree/master/jionlp/gadget"
    click R "https://github.com/geekywizkid/jionlp/tree/master/jionlp/rule"
    click T "https://github.com/geekywizkid/jionlp/tree/master/jionlp/textaug"
    click Uti "https://github.com/geekywizkid/jionlp/tree/master/jionlp/util"
    click Test "https://github.com/geekywizkid/jionlp/tree/master/test"
```
