# amzxyz/rime-lmdg architecture

> Architecture diagram of the GitHub repository amzxyz/rime-lmdg, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/amzxyz/rime-lmdg
- Repository: https://github.com/amzxyz/rime-lmdg

## Overview

#### Purpose and Deliverables
This repository builds and maintains a Chinese Rime input-method foundation: tone-marked pinyin lexicons, normalized word frequencies, pronunciation metadata, and downloadable grammar-model artifacts. Its primary consumable data is the Simplified/Traditional Rime dictionaries under `dicts/` and `dicts_hant/`, backed by comprehensive pinyin data in `pinyin_data/`.

#### Primary Entry Points
Batch-oriented Python scripts are the main operational interface:
- `wanxiang-tools.py` is the toolbox entry point for dictionary processing, pinyin refresh, auxiliary-code generation, short-code conversion, and user-dictionary work.
- `语法模型构建.py` is the model-building pipeline entry point.
- `run_core_tests.sh` and `run_core_tests.bat` provide cross-platform core validation.
- `rime-schema-compare/scripts/benchmark_sentences.py` runs decoder-based comparative evaluation.

#### Lexicon Products and Data Boundary
Rime YAML dictionaries are versioned output/data assets, partitioned by purpose rather than generated at runtime:
- Core/common and association dictionaries: `dicts/jichu.dict.yaml`, `dicts/lianxiang.dict.yaml`, `dicts/zi.dict.yaml`
- Specialized vocabularies: `dicts/diming.dict.yaml`, `dicts/yixue.dict.yaml`, `dicts/renming.dict.yaml`
- Traditional counterparts: `dicts_hant/jichu.dict.yaml`, `dicts_hant/zi.dict.yaml`

The central semantic boundary is a word entry coupled to tone-marked full pinyin and frequency. Simplified and Traditional lexicon trees are separate deployment targets.

#### Corpus-to-Lexicon/Model Pipeline
The build flow is offline and file-based: ingest raw corpora, clean/parse text, segment it, annotate pinyin, derive statistics, then filter/rank dictionary entries and construct grammar-model inputs. Key pipeline stages are represented by:
- Corpus cleanup and parsing: `TXT清洗.py`, `json语料解析.py`, `维基中文语料解析.py`
- Segmentation and scoring: `多线程分词.py`, `滑动窗口验证给短语打分并清洗无用词条.py`
- Frequency/model construction: `模型排序.py`, `语法模型构建.py`

Raw corpus scale and benchmark corpora are deliberately externalized from the main dictionaries; evaluation inputs live at `rime-schema-compare/data/corpus/`.

#### Pronunciation and Conversion Layer
The repository vendors a Python pinyin implementation and its curated data, enabling reproducible local annotation rather than relying on a network service:
- Pronunciation source data: `pinyin_data/单字.dict.yaml`, `pinyin_data/词组.dict.yaml`
- Pinyin engine: `pypinyin/core.py`, `pypinyin/converter.py`, `pypinyin/phrases_dict.json`
- Rime dictionary transforms: `rime固定或用户词典刷新为带声调编码.py`, `rime固定或用户词典刷新为带辅助码编码.py`

Supporting scripts bridge pinyin to input-schema encodings and synchronize user/Rime dictionaries: `scripts/pinyin2aux.py`, `scripts/sync_user_dict.py`, `scripts/deploy_rime.py`.

#### Rime Integration and Evaluation
The delivered dictionaries and grammar models are consumed by Rime-family runtimes (Weasel, Squirrel, or Fcitx5 with octagram support). The repository does not implement Rime itself; it interfaces with its decoder in the comparison subproject.

`rime-schema-compare/` is a Python benchmark harness that invokes native `librime`, runs schemas against corpora, computes sentence/character metrics, and emits reports:
- Native boundary: `rime-schema-compare/src/rime_schema_compare/call_librime.py`, `rime-schema-compare/lib/librime.1.16.1.dylib`
- Decode/evaluation orchestration: `rime-schema-compare/src/rime_schema_compare/rime_runner.py`, `rime-schema-compare/src/rime_schema_compare/metrics.py`
- Report generation: `rime-schema-compare/src/rime_schema_compare/reporting.py`, `rime-schema-compare/artifacts/benchmark_all_corpus_20260626T204334Z_report.txt`

#### Distribution and Auxiliary Clients
GitHub Actions automate nightly releases and toolbox workflows through `.github/workflows/nightly-release.yml` and `.github/workflows/wanxiang-tools.yml`. An independent Android/Kotlin updater client exists in `wanxiang-updater/`, with app startup in `wanxiang-updater/app/src/main/java/com/wanxiangupdater/MainActivity.kt`; it is a release/update consumer, not part of the lexicon build pipeline.

Main entry points: [`rime固定或用户词典刷新为带声调编码.py`](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/rime%E5%9B%BA%E5%AE%9A%E6%88%96%E7%94%A8%E6%88%B7%E8%AF%8D%E5%85%B8%E5%88%B7%E6%96%B0%E4%B8%BA%E5%B8%A6%E5%A3%B0%E8%B0%83%E7%BC%96%E7%A0%81.py) (Rime lexicon transforms), [`.github/workflows/nightly-release.yml`](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/.github/workflows/nightly-release.yml) (GitHub Actions releases), [`模型排序.py`](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/%E6%A8%A1%E5%9E%8B%E6%8E%92%E5%BA%8F.py) (Frequency ranking), [`pypinyin/core.py`](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/pypinyin/core.py) (Vendored pinyin engine)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_build["Offline build pipeline"]
  node_raw_corpora["Raw corpora<br/>external data"]
  node_corpus_parsing["Corpus cleanup &amp; parsing<br/>Python scripts<br/>[TXT清洗.py]"]
  node_segmentation["Segmentation &amp; phrase scoring<br/>Python scripts<br/>[多线程分词.py]"]
  node_frequency_ranking["Frequency ranking<br/>Python script<br/>[模型排序.py]"]
  node_pinyin_source["Curated pinyin data<br/>YAML lexicon data<br/>[单字.dict.yaml]"]
  node_pypinyin["Vendored pinyin engine<br/>Python library<br/>[core.py]"]
  node_lexicon_transforms["Rime lexicon transforms<br/>Python scripts"]
  node_grammar_builder["Grammar model builder<br/>Python pipeline<br/>[语法模型构建.py]"]
  node_toolbox["Dictionary toolbox<br/>Python CLI<br/>[wanxiang-tools.py]"]
end

subgraph group_lexicons["Rime data products"]
  node_grammar_models["Grammar-model artifacts<br/>model data"]
  node_simplified_dicts["Simplified dictionaries<br/>Rime YAML dictionaries<br/>[jichu.dict.yaml]"]
  node_traditional_dicts["Traditional dictionaries<br/>Rime YAML dictionaries<br/>[jichu.dict.yaml]"]
  node_rime_runtime{{"Rime-family runtime<br/>external decoder"}}
end

subgraph group_evaluation["Schema comparison"]
  node_benchmark_corpus["Benchmark corpora<br/>evaluation data"]
  node_benchmark_runner["Schema benchmark runner<br/>Python harness<br/>[rime_runner.py]"]
  node_librime{{"Native librime boundary<br/>native library bridge<br/>[call_librime.py]"}}
  node_metrics["Evaluation metrics<br/>Python module<br/>[metrics.py]"]
  node_benchmark_report["Benchmark reports<br/>report artifact"]
end

subgraph group_distribution["Distribution"]
  node_github_actions["GitHub Actions releases<br/>CI workflows"]
  node_android_updater["Android updater client<br/>Kotlin app<br/>[MainActivity.kt]"]
end

node_raw_corpora -->|"ingest"| node_corpus_parsing
node_corpus_parsing -->|"clean text"| node_segmentation
node_segmentation -->|"phrase statistics"| node_frequency_ranking
node_pinyin_source -->|"pronunciations"| node_pypinyin
node_frequency_ranking -->|"ranked entries"| node_lexicon_transforms
node_pypinyin -->|"tone-marked pinyin"| node_lexicon_transforms
node_lexicon_transforms -->|"write entries"| node_simplified_dicts
node_lexicon_transforms -->|"write entries"| node_traditional_dicts
node_frequency_ranking -->|"ranked corpus data"| node_grammar_builder
node_grammar_builder -->|"build"| node_grammar_models
node_toolbox -->|"refresh pinyin"| node_pypinyin
node_toolbox -->|"run transforms"| node_lexicon_transforms
node_simplified_dicts -->|"deploy"| node_rime_runtime
node_traditional_dicts -->|"deploy"| node_rime_runtime
node_grammar_models -->|"octagram input"| node_rime_runtime
node_benchmark_corpus -->|"sentences"| node_benchmark_runner
node_benchmark_runner -->|"decode requests"| node_librime
node_librime -->|"decoder results"| node_metrics
node_metrics -->|"generate report"| node_benchmark_report
node_github_actions -->|"nightly release"| node_simplified_dicts
node_github_actions -->|"publish"| node_grammar_models
node_github_actions -->|"release updates"| node_android_updater

click node_corpus_parsing "https://github.com/amzxyz/rime-lmdg/blob/wanxiang/TXT%E6%B8%85%E6%B4%97.py"
click node_segmentation "https://github.com/amzxyz/rime-lmdg/blob/wanxiang/%E5%A4%9A%E7%BA%BF%E7%A8%8B%E5%88%86%E8%AF%8D.py"
click node_frequency_ranking "https://github.com/amzxyz/rime-lmdg/blob/wanxiang/%E6%A8%A1%E5%9E%8B%E6%8E%92%E5%BA%8F.py"
click node_pinyin_source "https://github.com/amzxyz/rime-lmdg/blob/wanxiang/pinyin_data/%E5%8D%95%E5%AD%97.dict.yaml"
click node_pypinyin "https://github.com/amzxyz/rime-lmdg/blob/wanxiang/pypinyin/core.py"
click node_lexicon_transforms "https://github.com/amzxyz/rime-lmdg/blob/wanxiang/rime%E5%9B%BA%E5%AE%9A%E6%88%96%E7%94%A8%E6%88%B7%E8%AF%8D%E5%85%B8%E5%88%B7%E6%96%B0%E4%B8%BA%E5%B8%A6%E5%A3%B0%E8%B0%83%E7%BC%96%E7%A0%81.py"
click node_grammar_builder "https://github.com/amzxyz/rime-lmdg/blob/wanxiang/%E8%AF%AD%E6%B3%95%E6%A8%A1%E5%9E%8B%E6%9E%84%E5%BB%BA.py"
click node_simplified_dicts "https://github.com/amzxyz/rime-lmdg/blob/wanxiang/dicts/jichu.dict.yaml"
click node_traditional_dicts "https://github.com/amzxyz/rime-lmdg/blob/wanxiang/dicts_hant/jichu.dict.yaml"
click node_toolbox "https://github.com/amzxyz/rime-lmdg/blob/wanxiang/wanxiang-tools.py"
click node_benchmark_runner "https://github.com/amzxyz/rime-lmdg/blob/wanxiang/rime-schema-compare/src/rime_schema_compare/rime_runner.py"
click node_librime "https://github.com/amzxyz/rime-lmdg/blob/wanxiang/rime-schema-compare/src/rime_schema_compare/call_librime.py"
click node_metrics "https://github.com/amzxyz/rime-lmdg/blob/wanxiang/rime-schema-compare/src/rime_schema_compare/metrics.py"
click node_benchmark_report "https://github.com/amzxyz/rime-lmdg/blob/wanxiang/rime-schema-compare/artifacts/benchmark_all_corpus_20260626T204334Z_report.txt"
click node_github_actions "https://github.com/amzxyz/rime-lmdg/blob/wanxiang/.github/workflows/nightly-release.yml"
click node_android_updater "https://github.com/amzxyz/rime-lmdg/blob/wanxiang/wanxiang-updater/app/src/main/java/com/wanxiangupdater/MainActivity.kt"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_raw_corpora,node_corpus_parsing,node_segmentation,node_frequency_ranking,node_pinyin_source,node_pypinyin,node_lexicon_transforms,node_grammar_builder,node_toolbox toneBlue
class node_grammar_models,node_simplified_dicts,node_traditional_dicts,node_rime_runtime toneAmber
class node_benchmark_corpus,node_benchmark_runner,node_librime,node_metrics,node_benchmark_report toneMint
class node_github_actions,node_android_updater toneRose
```

## Components

### Offline build pipeline

File-based corpus and lexicon production.

- **Raw corpora**: external data · Large source corpora remain outside dictionary trees.
- [Corpus cleanup & parsing](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/TXT%E6%B8%85%E6%B4%97.py): Python scripts · Cleans text and parses JSON and Chinese Wikipedia corpora. · `TXT清洗.py`
- [Segmentation & phrase scoring](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/%E5%A4%9A%E7%BA%BF%E7%A8%8B%E5%88%86%E8%AF%8D.py): Python scripts · Segments corpus text and validates or scores phrases. · `多线程分词.py`
- [Frequency ranking](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/%E6%A8%A1%E5%9E%8B%E6%8E%92%E5%BA%8F.py): Python script · Normalizes and ranks lexical statistics. · `模型排序.py`
- [Curated pinyin data](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/pinyin_data/%E5%8D%95%E5%AD%97.dict.yaml): YAML lexicon data · Single-character and phrase pronunciation sources. · `pinyin_data/单字.dict.yaml`
- [Vendored pinyin engine](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/pypinyin/core.py): Python library · Locally annotates words with tone-marked pinyin. · `pypinyin/core.py`
- [Rime lexicon transforms](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/rime%E5%9B%BA%E5%AE%9A%E6%88%96%E7%94%A8%E6%88%B7%E8%AF%8D%E5%85%B8%E5%88%B7%E6%96%B0%E4%B8%BA%E5%B8%A6%E5%A3%B0%E8%B0%83%E7%BC%96%E7%A0%81.py): Python scripts · Refreshes tone-marked encodings and auxiliary codes. · `rime固定或用户词典刷新为带声调编码.py`
- [Grammar model builder](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/%E8%AF%AD%E6%B3%95%E6%A8%A1%E5%9E%8B%E6%9E%84%E5%BB%BA.py): Python pipeline · Constructs grammar-model inputs from ranked corpus data. · `语法模型构建.py`
- [Dictionary toolbox](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/wanxiang-tools.py): Python CLI · Operational entry point for dictionary, pinyin, code, and user-dictionary tasks. · `wanxiang-tools.py`

### Rime data products

Separate Simplified and Traditional deployment trees.

- **Grammar-model artifacts**: model data · Downloadable artifacts for octagram-capable Rime runtimes.
- [Simplified dictionaries](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/dicts/jichu.dict.yaml): Rime YAML dictionaries · Core, association, character, and specialized vocabulary. · `dicts/jichu.dict.yaml`
- [Traditional dictionaries](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/dicts_hant/jichu.dict.yaml): Rime YAML dictionaries · Traditional deployment counterpart lexicons. · `dicts_hant/jichu.dict.yaml`
- **Rime-family runtime**: external decoder · Weasel, Squirrel, or Fcitx5 consumes deployed dictionaries and models.

### Schema comparison

Decoder-based benchmark harness.

- **Benchmark corpora**: evaluation data · Externalized sentence inputs for schema comparisons.
- [Schema benchmark runner](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/rime-schema-compare/src/rime_schema_compare/rime_runner.py): Python harness · Runs schemas through the native decoder. · `rime-schema-compare/src/rime_schema_compare/rime_runner.py`
- [Native librime boundary](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/rime-schema-compare/src/rime_schema_compare/call_librime.py): native library bridge · Python binding invokes the Rime decoder. · `rime-schema-compare/src/rime_schema_compare/call_librime.py`
- [Evaluation metrics](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/rime-schema-compare/src/rime_schema_compare/metrics.py): Python module · Computes sentence and character metrics. · `rime-schema-compare/src/rime_schema_compare/metrics.py`
- [Benchmark reports](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/rime-schema-compare/artifacts/benchmark_all_corpus_20260626T204334Z_report.txt): report artifact · Comparison results emitted as text reports. · `rime-schema-compare/artifacts/benchmark_all_corpus_20260626T204334Z_report.txt`

### Distribution

- [GitHub Actions releases](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/.github/workflows/nightly-release.yml): CI workflows · Automates nightly releases and toolbox workflows. · `.github/workflows/nightly-release.yml`
- [Android updater client](https://github.com/amzxyz/rime-lmdg/blob/wanxiang/wanxiang-updater/app/src/main/java/com/wanxiangupdater/MainActivity.kt): Kotlin app · Independent release and update consumer. · `wanxiang-updater/app/src/main/java/com/wanxiangupdater/MainActivity.kt`

## Connections

- Raw corpora → Corpus cleanup & parsing: ingest
- Corpus cleanup & parsing → Segmentation & phrase scoring: clean text
- Segmentation & phrase scoring → Frequency ranking: phrase statistics
- Curated pinyin data → Vendored pinyin engine: pronunciations
- Frequency ranking → Rime lexicon transforms: ranked entries
- Vendored pinyin engine → Rime lexicon transforms: tone-marked pinyin
- Rime lexicon transforms → Simplified dictionaries: write entries
- Rime lexicon transforms → Traditional dictionaries: write entries
- Frequency ranking → Grammar model builder: ranked corpus data
- Grammar model builder → Grammar-model artifacts: build
- Dictionary toolbox → Vendored pinyin engine: refresh pinyin
- Dictionary toolbox → Rime lexicon transforms: run transforms
- Simplified dictionaries → Rime-family runtime: deploy
- Traditional dictionaries → Rime-family runtime: deploy
- Grammar-model artifacts → Rime-family runtime: octagram input
- Benchmark corpora → Schema benchmark runner: sentences
- Schema benchmark runner → Native librime boundary: decode requests
- Native librime boundary → Evaluation metrics: decoder results
- Evaluation metrics → Benchmark reports: generate report
- GitHub Actions releases → Simplified dictionaries: nightly release
- GitHub Actions releases → Grammar-model artifacts: publish
- GitHub Actions releases → Android updater client: release updates