Install & Compatibility
Where this runs
tested against v3.1.1 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
85MB installed
● package 85MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
word_frequency
✓ from wordfreq import word_frequency
zipf_frequency
✓ from wordfreq import zipf_frequency
tokenize
✓ from wordfreq import tokenize
✗ from wordfreq.preprocess import tokenize
The `tokenize` function moved to the top-level `wordfreq` module in v2.0, with `preprocess.preprocess_text` and `lossy_tokenize` introduced for specific preprocessing steps.
This quickstart demonstrates how to use the `word_frequency` and `zipf_frequency` functions to retrieve word frequencies in different languages and scales. `word_frequency` returns a decimal between 0 and 1, while `zipf_frequency` returns a value on a human-friendly logarithmic scale.
from wordfreq import word_frequency, zipf_frequency
# Get the raw frequency (between 0 and 1)
freq_en = word_frequency('the', 'en')
print(f"Frequency of 'the' in English: {freq_en}")
freq_fr_cafe = word_frequency('café', 'fr')
print(f"Frequency of 'café' in French: {freq_fr_cafe}")
# Get the Zipf frequency (logarithmic scale, base-10 logarithm of occurrences per billion words)
zipf_en = zipf_frequency('computer', 'en')
print(f"Zipf frequency of 'computer' in English: {zipf_en}")
zipf_nonexistent = zipf_frequency('nonexistentword123', 'en')
print(f"Zipf frequency of 'nonexistentword123' in English: {zipf_nonexistent}")
# Example with a different wordlist (default is 'best', 'large' or 'small' can be specified)
zipf_large = zipf_frequency('quantum', 'en', wordlist='large')
print(f"Zipf frequency of 'quantum' (large list) in English: {zipf_large}")
Debug
Known issues
breakingIn version 3.0, the handling of multi-digit numbers changed. Previously, sequences of two or more digits were grouped into a single token (e.g., '1234' became '0000'), leading to an overestimated frequency. Now, frequencies are distributed across numbers of that shape, incorporating Benford's law and special handling for 4-digit years, providing more realistic estimates. Functions like `iter_wordlist` and `top_n_list` also no longer return multi-digit numbers.fixBe aware that number-containing words may yield different frequencies compared to pre-3.0 versions. Review your logic if your application relies on specific numeric token representations.
affects: >=3.0
breakingVersion 3.0 (building on changes from 2.0) significantly altered tokenization functions. The `tokenize` function no longer supports a `combine_numbers` option (which was implicitly removed as `lossy_tokenize` provides similar behavior for combining numbers). Additionally, `tokenize` no longer automatically replaces Chinese characters with their Simplified Chinese versions; this transformation is now handled by `lossy_tokenize`.fixIf you require specific preprocessing steps such as combining numbers or Chinese character simplification, explicitly use `wordfreq.lossy_tokenize(text, lang)` instead of `wordfreq.tokenize(text, lang)`.
affects: >=3.0
gotchaWordfreq relies on the `regex` library for tokenization. Versions of `regex` prior to `2021.7.6` do not include the `regex.Match` class, which can lead to import errors or unexpected behavior. Ensure your `regex` installation is up-to-date.fixUpgrade the `regex` library to version `2021.7.6` or newer: `pip install --upgrade regex`.
affects: <2021.7.6 of `regex` dependency
gotchaThe word frequency data provided by `wordfreq` is based on language usage up to approximately 2021 and will not be updated further. This decision was made because generative AI models have 'polluted' online data sources, making it difficult to obtain reliable information about post-2021 human language usage.fixBe aware that `wordfreq` reflects historical language usage. For analyses requiring current, post-2021 language trends, `wordfreq`'s data may not be representative. Consider alternative methods or acknowledge this data limitation.
affects: All versions, regarding data content
gotchaSupport for Chinese, Japanese, and Korean (CJK) languages requires additional optional dependencies (`jieba`, `mecab-python3`, `ipadic`, `mecab-ko-dic`). For Japanese and Korean tokenization using `mecab-python3`, you may also need to install the `libmecab-dev` system package, which can be complex depending on your operating system.fixInstall CJK dependencies using `pip install wordfreq[cjk]`. For MeCab (Japanese/Korean), consult `mecab-python3` documentation for system-level prerequisites like `libmecab-dev`.
affects: All versions, for CJK language support
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'wordfreq'
The `wordfreq` library has not been installed in the current Python environment.
KeyError: 'no such language: de'
The requested language code (e.g., 'de') is not supported by the installed `wordfreq` library.
fixCheck available languages using `wordfreq.get_supported_languages()` and use a supported code.
ValueError: The input text must be a string.
A non-string value was passed as the `word` argument to `wordfreq.word_frequency()` or `wordfreq.zipf_frequency()`.
fixEnsure the first argument passed to `wordfreq.word_frequency()` or `wordfreq.zipf_frequency()` is a string.
AttributeError: module 'wordfreq' has no attribute 'frequency'
The function name `frequency` is incorrect; the correct function for word frequency is `word_frequency` or `zipf_frequency`.
fixUse `wordfreq.word_frequency(word, lang)` for raw frequency or `wordfreq.zipf_frequency(word, lang)` for Zipf-scale frequency.
Upgrade
Version history
3.1.1latest on PyPI · released Nov 21, 2023
Audit
Dependencies
msgpackrequiredCore dependency for data handling.
langcodesrequiredCore dependency for language code handling.
regexrequiredCore dependency for tokenization; requires specific version for `regex.Match` class.
jiebaoptionalOptional dependency for Chinese language tokenization.
mecab-python3optionalOptional dependency for Japanese and Korean language tokenization. Requires system package `libmecab-dev`.
ipadicoptionalOptional dependency for Japanese language tokenization (used with `mecab-python3`).
mecab-ko-dicoptionalOptional dependency for Korean language tokenization (used with `mecab-python3`).