How atilika/kuromoji works
Main Components
• Core Library (kuromoji-core): This holds the fundamental classes and utility methods (e.g., TokenBase, TokenizerBase, buffer classes, FST implementations, dictionary compiling logic). It is the “engine” that other modules extend.
• Dictionary Modules: Each dictionary has its own Maven module (e.g., kuromoji-ipadic, kuromoji-ipadic-neologd, kuromoji-jumandic, kuromoji-naist-jdic, kuromoji-unidic, etc.). They provide specialized implementations and token classes (Tokenizer, Token) in distinct Java packages tailored to the dictionary.
• Benchmark Module (kuromoji-benchmark): This module contains performance testing utilities (for example, Benchmark, HeapUtilizationSnapshot, UserDictionaryConverter, etc.) to help benchmark the performance of tokenizers.
• Compiler & Builder Components: Within the core and dictionary modules there are components responsible for compiling dictionaries (e.g., FSTCompiler, DictionaryCompiler, TokenInfoBufferCompiler) which transform raw dictionary data into efficient lookup structures.
Relationships and Interactions
• The specialized Tokenizer and Token classes in each dictionary module implement interfaces or extend base classes defined in the core module. Represent these relationships by linking the dictionary Tokenizers back to the core’s TokenizerBase.
• The Benchmark module interacts with both the core and dictionary modules by importing them to run synthetic tests and performance benchmarks.
• The compilation components (often found under packages like “compile” in the core and dictionary modules) interact with raw dictionary inputs to produce binary files or lookup structures used at runtime by the morphological analyzers.
Architectural Patterns and Design Principles
• Modular Architecture: The project is divided into discrete modules, each with a precise responsibility (core functionality, dictionary-specific features, benchmarking).
• Dependency Management: The use of Maven’s multi-module project structure ensures that each module is developed, compiled, and tested in isolation but then integrated during build and packaging.
• Separation of Concerns: The core module handles generic analysis and resource management, while dictionary modules encapsulate dictionary-specific logic.
• Extensibility: By isolating different dictionaries into separate modules, the design ensures that adding or updating a dictionary module does not disrupt the rest of the system.
Relevant Technologies, Frameworks, and Libraries
• Java (with Maven as the build tool)
• JUnit (for unit testing, as seen in the test structure)
• Apache Maven (for dependency and build management)
• Travis CI (indicated via .travis.yml – for continuous integration)