# rspeer/wordfreq architecture

> Architecture diagram of the GitHub repository rspeer/wordfreq, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rspeer/wordfreq
- Repository: https://github.com/rspeer/wordfreq

## Overview

wordfreq is a multilingual Python lookup library: callers submit a word and language, the library standardizes and tokenizes it, handles numeric sequences specially, consults compact per-language frequency lists, and returns either proportional or Zipf-scale frequency. It also exposes wordlist iteration/top-N and random-word APIs. The graph follows the sampled lookup and tokenization code; broader corpus-source combining is documented as upstream provenance, not a runtime workflow in this repository.

Main entry points: [`wordfreq/__init__.py`](https://github.com/rspeer/wordfreq/blob/master/wordfreq/__init__.py) (Lookup and list API), [`wordfreq/tokens.py`](https://github.com/rspeer/wordfreq/blob/master/wordfreq/tokens.py) (Tokenization), [`wordfreq/preprocess.py`](https://github.com/rspeer/wordfreq/blob/master/wordfreq/preprocess.py) (Text normalization), [`wordfreq/chinese.py`](https://github.com/rspeer/wordfreq/blob/master/wordfreq/chinese.py) (Chinese handling)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Lookup API"]
  node_api["Lookup and list API<br/>[__init__.py]"]
end

subgraph group_text["Text Processing"]
  node_metadata["Language metadata<br/>[language_info.py]"]
  node_tokens["Tokenization<br/>[tokens.py]"]
  node_preprocess["Text normalization<br/>[preprocess.py]"]
  node_translit["Script transliteration<br/>[transliterate.py]"]
  node_cjk["Chinese handling<br/>[chinese.py]"]
  node_mecab["Japanese/Korean tokenizer<br/>[mecab.py]"]
  node_numbers["Number frequency rules<br/>[numbers.py]"]
end

subgraph group_data["Frequency Data"]
  node_freqdata["Compressed wordlists"]
  node_datautil["Data path resolution<br/>[util.py]"]
end

node_caller(("Library caller"))

node_caller -->|"calls lookup"| node_api
node_api -->|"gets language rules"| node_metadata
node_api -->|"tokenizes input"| node_tokens
node_tokens -->|"selects tokenizer"| node_metadata
node_tokens -->|"normalizes text"| node_preprocess
node_preprocess -->|"reads language rules"| node_metadata
node_preprocess -->|"transliterates scripts"| node_translit
node_tokens -.->|"uses Chinese tokenizer"| node_cjk
node_tokens -.->|"uses CJK tokenizer"| node_mecab
node_api -->|"normalizes digit sequences"| node_numbers
node_api -->|"loads frequency bins"| node_freqdata
node_api -->|"resolves data directory"| node_datautil
node_cjk -->|"resolves dictionary data"| node_datautil
node_cjk -->|"loads Chinese mapping"| node_freqdata

click node_api "https://github.com/rspeer/wordfreq/blob/master/wordfreq/__init__.py"
click node_metadata "https://github.com/rspeer/wordfreq/blob/master/wordfreq/language_info.py"
click node_tokens "https://github.com/rspeer/wordfreq/blob/master/wordfreq/tokens.py"
click node_preprocess "https://github.com/rspeer/wordfreq/blob/master/wordfreq/preprocess.py"
click node_translit "https://github.com/rspeer/wordfreq/blob/master/wordfreq/transliterate.py"
click node_cjk "https://github.com/rspeer/wordfreq/blob/master/wordfreq/chinese.py"
click node_mecab "https://github.com/rspeer/wordfreq/blob/master/wordfreq/mecab.py"
click node_numbers "https://github.com/rspeer/wordfreq/blob/master/wordfreq/numbers.py"
click node_freqdata "https://github.com/rspeer/wordfreq/tree/master/wordfreq/data"
click node_datautil "https://github.com/rspeer/wordfreq/blob/master/wordfreq/util.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api toneBlue
class node_metadata,node_tokens,node_preprocess,node_translit,node_cjk,node_mecab,node_numbers toneAmber
class node_freqdata,node_datautil toneMint
class node_caller toneIndigo
```

## Components

### Lookup API

- [Lookup and list API](https://github.com/rspeer/wordfreq/blob/master/wordfreq/__init__.py): `wordfreq/__init__.py`

### Text Processing

- [Language metadata](https://github.com/rspeer/wordfreq/blob/master/wordfreq/language_info.py): `wordfreq/language_info.py`
- [Tokenization](https://github.com/rspeer/wordfreq/blob/master/wordfreq/tokens.py): `wordfreq/tokens.py`
- [Text normalization](https://github.com/rspeer/wordfreq/blob/master/wordfreq/preprocess.py): `wordfreq/preprocess.py`
- [Script transliteration](https://github.com/rspeer/wordfreq/blob/master/wordfreq/transliterate.py): `wordfreq/transliterate.py`
- [Chinese handling](https://github.com/rspeer/wordfreq/blob/master/wordfreq/chinese.py): `wordfreq/chinese.py`
- [Japanese/Korean tokenizer](https://github.com/rspeer/wordfreq/blob/master/wordfreq/mecab.py): `wordfreq/mecab.py`
- [Number frequency rules](https://github.com/rspeer/wordfreq/blob/master/wordfreq/numbers.py): `wordfreq/numbers.py`

### Frequency Data

- [Compressed wordlists](https://github.com/rspeer/wordfreq/tree/master/wordfreq/data): `wordfreq/data`
- [Data path resolution](https://github.com/rspeer/wordfreq/blob/master/wordfreq/util.py): `wordfreq/util.py`

### Other components

- **Library caller**

## Connections

- Library caller → Lookup and list API: calls lookup
- Lookup and list API → Language metadata: gets language rules
- Lookup and list API → Tokenization: tokenizes input
- Tokenization → Language metadata: selects tokenizer
- Tokenization → Text normalization: normalizes text
- Text normalization → Language metadata: reads language rules
- Text normalization → Script transliteration: transliterates scripts
- Tokenization → Chinese handling: uses Chinese tokenizer
- Tokenization → Japanese/Korean tokenizer: uses CJK tokenizer
- Lookup and list API → Number frequency rules: normalizes digit sequences
- Lookup and list API → Compressed wordlists: loads frequency bins
- Lookup and list API → Data path resolution: resolves data directory
- Chinese handling → Data path resolution: resolves dictionary data
- Chinese handling → Compressed wordlists: loads Chinese mapping