amzxyz/rime-lmdg

Purpose and Deliverables

This repository builds and maintains a Chinese Rime input-method foundation: tone-marked pinyin lexicons, normalized word frequencies, pronunciation metadata, and downloadable grammar-model artifacts. Its primary consumable data is the Simplified/Traditional Rime dictionaries under dicts/ and dicts_hant/, backed by comprehensive pinyin data in pinyin_data/.

Primary Entry Points

Batch-oriented Python scripts are the main operational interface:

• wanxiang-tools.py is the toolbox entry point for dictionary processing, pinyin refresh, auxiliary-code generation, short-code conversion, and user-dictionary work.

• 语法模型构建.py is the model-building pipeline entry point.

• run_core_tests.sh and run_core_tests.bat provide cross-platform core validation.

• rime-schema-compare/scripts/benchmark_sentences.py runs decoder-based comparative evaluation.

Lexicon Products and Data Boundary

Rime YAML dictionaries are versioned output/data assets, partitioned by purpose rather than generated at runtime:

• Core/common and association dictionaries: dicts/jichu.dict.yaml, dicts/lianxiang.dict.yaml, dicts/zi.dict.yaml

• Specialized vocabularies: dicts/diming.dict.yaml, dicts/yixue.dict.yaml, dicts/renming.dict.yaml

• Traditional counterparts: dicts_hant/jichu.dict.yaml, dicts_hant/zi.dict.yaml

The central semantic boundary is a word entry coupled to tone-marked full pinyin and frequency. Simplified and Traditional lexicon trees are separate deployment targets.

Corpus-to-Lexicon/Model Pipeline

The build flow is offline and file-based: ingest raw corpora, clean/parse text, segment it, annotate pinyin, derive statistics, then filter/rank dictionary entries and construct grammar-model inputs. Key pipeline stages are represented by:

• Corpus cleanup and parsing: TXT清洗.py, json语料解析.py, 维基中文语料解析.py

• Segmentation and scoring: 多线程分词.py, 滑动窗口验证给短语打分并清洗无用词条.py

• Frequency/model construction: 模型排序.py, 语法模型构建.py

Raw corpus scale and benchmark corpora are deliberately externalized from the main dictionaries; evaluation inputs live at rime-schema-compare/data/corpus/.

Pronunciation and Conversion Layer

The repository vendors a Python pinyin implementation and its curated data, enabling reproducible local annotation rather than relying on a network service:

• Pronunciation source data: pinyin_data/单字.dict.yaml, pinyin_data/词组.dict.yaml

• Pinyin engine: pypinyin/core.py, pypinyin/converter.py, pypinyin/phrases_dict.json

• Rime dictionary transforms: rime固定或用户词典刷新为带声调编码.py, rime固定或用户词典刷新为带辅助码编码.py

Supporting scripts bridge pinyin to input-schema encodings and synchronize user/Rime dictionaries: scripts/pinyin2aux.py, scripts/sync_user_dict.py, scripts/deploy_rime.py.

Rime Integration and Evaluation

The delivered dictionaries and grammar models are consumed by Rime-family runtimes (Weasel, Squirrel, or Fcitx5 with octagram support). The repository does not implement Rime itself; it interfaces with its decoder in the comparison subproject.

rime-schema-compare/ is a Python benchmark harness that invokes native librime, runs schemas against corpora, computes sentence/character metrics, and emits reports:

• Native boundary: rime-schema-compare/src/rime_schema_compare/call_librime.py, rime-schema-compare/lib/librime.1.16.1.dylib

• Decode/evaluation orchestration: rime-schema-compare/src/rime_schema_compare/rime_runner.py, rime-schema-compare/src/rime_schema_compare/metrics.py

• Report generation: rime-schema-compare/src/rime_schema_compare/reporting.py, rime-schema-compare/artifacts/benchmark_all_corpus_20260626T204334Z_report.txt

Distribution and Auxiliary Clients

GitHub Actions automate nightly releases and toolbox workflows through .github/workflows/nightly-release.yml and .github/workflows/wanxiang-tools.yml. An independent Android/Kotlin updater client exists in wanxiang-updater/, with app startup in wanxiang-updater/app/src/main/java/com/wanxiangupdater/MainActivity.kt; it is a release/update consumer, not part of the lexicon build pipeline.