How rspeer/wordfreq works
wordfreq is a multilingual Python lookup library: callers submit a word and language, the library standardizes and tokenizes it, handles numeric sequences specially, consults compact per-language frequency lists, and returns either proportional or Zipf-scale frequency. It also exposes wordlist iteration/top-N and random-word APIs. The graph follows the sampled lookup and tokenization code; broader corpus-source combining is documented as upstream provenance, not a runtime workflow in this repository.
Main entry points: __init__.py Lookup and list API · tokens.py Tokenization · preprocess.py Text normalization · chinese.py Chinese handling
Components 11
Lookup API
- Lookup and list API
wordfreq/__init__.py
Text Processing
- Language metadata
wordfreq/language_info.py - Tokenization
wordfreq/tokens.py - Text normalization
wordfreq/preprocess.py - Script transliteration
wordfreq/transliterate.py - Chinese handling
wordfreq/chinese.py - Japanese/Korean tokenizer
wordfreq/mecab.py - Number frequency rules
wordfreq/numbers.py
Frequency Data
- Compressed wordlists
wordfreq/data - Data path resolution
wordfreq/util.py
Other components
- Library caller
Connections 14
- Library caller → Lookup and list API: calls lookup
- Lookup and list API → Language metadata: gets language rules
- Lookup and list API → Tokenization: tokenizes input
- Tokenization → Language metadata: selects tokenizer
- Tokenization → Text normalization: normalizes text
- Text normalization → Language metadata: reads language rules
- Text normalization → Script transliteration: transliterates scripts
- Tokenization → Chinese handling: uses Chinese tokenizer
- Tokenization → Japanese/Korean tokenizer: uses CJK tokenizer
- Lookup and list API → Number frequency rules: normalizes digit sequences
- Lookup and list API → Compressed wordlists: loads frequency bins
- Lookup and list API → Data path resolution: resolves data directory
- Chinese handling → Data path resolution: resolves dictionary data
- Chinese handling → Compressed wordlists: loads Chinese mapping