# atilika/kuromoji architecture

> Architecture diagram of the GitHub repository atilika/kuromoji, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/atilika/kuromoji
- Repository: https://github.com/atilika/kuromoji

## Overview

##### Main Components
- Core Library (kuromoji-core): This holds the fundamental classes and utility methods (e.g., TokenBase, TokenizerBase, buffer classes, FST implementations, dictionary compiling logic). It is the “engine” that other modules extend.
- Dictionary Modules: Each dictionary has its own Maven module (e.g., kuromoji-ipadic, kuromoji-ipadic-neologd, kuromoji-jumandic, kuromoji-naist-jdic, kuromoji-unidic, etc.). They provide specialized implementations and token classes (Tokenizer, Token) in distinct Java packages tailored to the dictionary.
- Benchmark Module (kuromoji-benchmark): This module contains performance testing utilities (for example, Benchmark, HeapUtilizationSnapshot, UserDictionaryConverter, etc.) to help benchmark the performance of tokenizers.
- Compiler & Builder Components: Within the core and dictionary modules there are components responsible for compiling dictionaries (e.g., FSTCompiler, DictionaryCompiler, TokenInfoBufferCompiler) which transform raw dictionary data into efficient lookup structures.

##### Relationships and Interactions
- The specialized Tokenizer and Token classes in each dictionary module implement interfaces or extend base classes defined in the core module. Represent these relationships by linking the dictionary Tokenizers back to the core’s TokenizerBase.
- The Benchmark module interacts with both the core and dictionary modules by importing them to run synthetic tests and performance benchmarks.
- The compilation components (often found under packages like “compile” in the core and dictionary modules) interact with raw dictionary inputs to produce binary files or lookup structures used at runtime by the morphological analyzers.

##### Architectural Patterns and Design Principles
- Modular Architecture: The project is divided into discrete modules, each with a precise responsibility (core functionality, dictionary-specific features, benchmarking).
- Dependency Management: The use of Maven’s multi-module project structure ensures that each module is developed, compiled, and tested in isolation but then integrated during build and packaging.
- Separation of Concerns: The core module handles generic analysis and resource management, while dictionary modules encapsulate dictionary-specific logic.
- Extensibility: By isolating different dictionaries into separate modules, the design ensures that adding or updating a dictionary module does not disrupt the rest of the system.

##### Relevant Technologies, Frameworks, and Libraries
- Java (with Maven as the build tool)
- JUnit (for unit testing, as seen in the test structure)
- Apache Maven (for dependency and build management)
- Travis CI (indicated via .travis.yml – for continuous integration)

## Diagram (Mermaid)

```mermaid
flowchart TD

    %% Build & Deployment
    subgraph "Build & Deployment"
        Maven["Maven Build"]:::build
    end

    %% Core Layer
    subgraph "Core Layer"
        Core["Kuromoji Core"]:::core
        CoreComp["Core Compiler"]:::compiler
    end

    %% Dictionary Modules
    subgraph "Dictionary Modules"
        IPADIC["IPADIC"]:::dictionary
        IPADICComp["IPADIC Compiler"]:::compiler

        IPADICN["IPADIC Neologd"]:::dictionary
        IPADICNComp["IPADIC Neologd Compiler"]:::compiler

        JUMANDIC["JUMANDIC"]:::dictionary
        JUMANDICComp["JUMANDIC Compiler"]:::compiler

        NAIST["NAIST jdic"]:::dictionary
        NAISTComp["NAIST jdic Compiler"]:::compiler

        UNIDIC["UniDic"]:::dictionary
        UNIDICComp["UniDic Compiler"]:::compiler

        UNIKANA["UniDic KanaAccent"]:::dictionary
        UNIKANAComp["UniDic KanaAccent Compiler"]:::compiler

        UNINEO["UniDic Neologd"]:::dictionary
        UNINEOComp["UniDic Neologd Compiler"]:::compiler
    end

    %% Benchmarking Module
    subgraph "Benchmarking"
        Benchmark["Kuromoji Benchmark"]:::benchmark
    end

    %% Relationships: Build & Deployment to Modules
    Maven --> Core
    Maven --> IPADIC
    Maven --> IPADICN
    Maven --> JUMANDIC
    Maven --> NAIST
    Maven --> UNIDIC
    Maven --> UNIKANA
    Maven --> UNINEO
    Maven --> Benchmark

    %% Core Compiler
    Core -->|"compiles"| CoreComp

    %% Dictionary Modules Compilation relationships
    IPADIC -->|"compiles"| IPADICComp
    IPADICN -->|"compiles"| IPADICNComp
    JUMANDIC -->|"compiles"| JUMANDICComp
    NAIST -->|"compiles"| NAISTComp
    UNIDIC -->|"compiles"| UNIDICComp
    UNIKANA -->|"compiles"| UNIKANAComp
    UNINEO -->|"compiles"| UNINEOComp

    %% Dictionary Modules depend on Core
    IPADIC -->|"uses"| Core
    IPADICN -->|"uses"| Core
    JUMANDIC -->|"uses"| Core
    NAIST -->|"uses"| Core
    UNIDIC -->|"uses"| Core
    UNIKANA -->|"uses"| Core
    UNINEO -->|"uses"| Core

    %% Benchmark Module interactions
    Benchmark -->|"tests"| Core
    Benchmark -->|"benchmarks"| IPADIC
    Benchmark -->|"benchmarks"| IPADICN
    Benchmark -->|"benchmarks"| JUMANDIC
    Benchmark -->|"benchmarks"| NAIST
    Benchmark -->|"benchmarks"| UNIDIC
    Benchmark -->|"benchmarks"| UNIKANA
    Benchmark -->|"benchmarks"| UNINEO

    %% Click Events
    click Core "https://github.com/atilika/kuromoji/tree/master/kuromoji-core/src/main/java/com/atilika/kuromoji"
    click IPADIC "https://github.com/atilika/kuromoji/tree/master/kuromoji-ipadic/src/main/java/com/atilika/kuromoji/ipadic"
    click IPADICN "https://github.com/atilika/kuromoji/tree/master/kuromoji-ipadic-neologd/src/main/java/com/atilika/kuromoji/ipadic/neologd"
    click JUMANDIC "https://github.com/atilika/kuromoji/tree/master/kuromoji-jumandic/src/main/java/com/atilika/kuromoji/jumandic"
    click NAIST "https://github.com/atilika/kuromoji/tree/master/kuromoji-naist-jdic/src/main/java/com/atilika/kuromoji/naist/jdic"
    click UNIDIC "https://github.com/atilika/kuromoji/tree/master/kuromoji-unidic/src/main/java/com/atilika/kuromoji/unidic"
    click UNIKANA "https://github.com/atilika/kuromoji/tree/master/kuromoji-unidic-kanaaccent/src/main/java/com/atilika/kuromoji/unidic/kanaaccent"
    click UNINEO "https://github.com/atilika/kuromoji/tree/master/kuromoji-unidic-neologd/src/main/java/com/atilika/kuromoji/unidic/neologd"
    click Benchmark "https://github.com/atilika/kuromoji/tree/master/kuromoji-benchmark/src/main/java/com/atilika/kuromoji/benchmark"
    click CoreComp "https://github.com/atilika/kuromoji/tree/master/kuromoji-core/src/main/java/com/atilika/kuromoji/compile"
    click IPADICComp "https://github.com/atilika/kuromoji/tree/master/kuromoji-ipadic/src/main/java/com/atilika/kuromoji/ipadic/compile"
    click IPADICNComp "https://github.com/atilika/kuromoji/tree/master/kuromoji-ipadic-neologd/src/main/java/com/atilika/kuromoji/ipadic/neologd/compile"
    click JUMANDICComp "https://github.com/atilika/kuromoji/tree/master/kuromoji-jumandic/src/main/java/com/atilika/kuromoji/jumandic/compile"
    click NAISTComp "https://github.com/atilika/kuromoji/tree/master/kuromoji-naist-jdic/src/main/java/com/atilika/kuromoji/naist/jdic/compile"
    click UNIDICComp "https://github.com/atilika/kuromoji/tree/master/kuromoji-unidic/src/main/java/com/atilika/kuromoji/unidic/compile"
    click UNIKANAComp "https://github.com/atilika/kuromoji/tree/master/kuromoji-unidic-kanaaccent/src/main/java/com/atilika/kuromoji/unidic/kanaaccent/compile"
    click UNINEOComp "https://github.com/atilika/kuromoji/tree/master/kuromoji-unidic-neologd/src/main/java/com/atilika/kuromoji/unidic/neologd/compile"

    %% Styles
    classDef core fill:#A2D2FF,stroke:#333,stroke-width:2px;
    classDef dictionary fill:#FFC9DE,stroke:#333,stroke-width:2px;
    classDef compiler fill:#CFFFB3,stroke:#333,stroke-width:2px;
    classDef benchmark fill:#FDE68A,stroke:#333,stroke-width:2px;
    classDef build fill:#D1FAE5,stroke:#333,stroke-width:2px;
```
