Registry / ai-ml / wetextprocessing

wetextprocessing

library1.2.0pypypi✓ verified 86d ago

WeTextProcessing is an active Python library providing production-ready Text Normalization (TN) and Inverse Text Normalization (ITN) capabilities. It primarily supports Chinese, English, and Japanese languages, leveraging Finite State Transducers (FSTs) for efficient processing. The library has a consistent release cadence, with multiple minor updates released throughout 2024 to introduce new features, improvements, and bug fixes.

ai-ml data

pip install WeTextProcessing

github ↗package ↗

INSTALL

IMPORT

SIG · WETEXTPROCESSING

wetextprocessing

ai-mlpythonv1.2.0

Install

9.4s avg

Import

120ms

Disk

739MB

Pass rate

4/ 10

Env Coverage4 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v1.2.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

glibc

py 3.10

✕ build_error

✓ 6.5s

py 3.11

✕ build_error

✓ 6.6s

py 3.12

✕ build_error

✓ 5.93s

py 3.13

✕ build_error

1/4 runs

py 3.9

✕ build_error

✓ 18.68s

739MB installed

● package 739MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

Normalizer (Chinese TN)

✓ from tn.chinese.normalizer import Normalizer

✗ from wetextprocessing import Normalizer

For `WeTextProcessing` (the pypi-slug), specific language normalizers are imported from `tn.<lang>.normalizer` or `itn.<lang>.inverse_normalizer`. The `wetext` package uses a different top-level `Normalizer` import.

InverseNormalizer (Chinese ITN)

✓ from itn.chinese.inverse_normalizer import InverseNormalizer

✗ from wetextprocessing import InverseNormalizer

Inverse Normalizer for Chinese is specifically imported from its language-specific path within the `itn` submodule.

Normalizer (English TN)

✓ from tn.english.normalizer import Normalizer as EnNormalizer

✗ from tn.chinese.normalizer import Normalizer

For English Text Normalization, use the specific English normalizer.

This quickstart demonstrates how to perform Chinese Text Normalization (TN), Chinese Inverse Text Normalization (ITN), and English Text Normalization using the `WeTextProcessing` library. It showcases specific imports for each language and the use of `overwrite_cache=True` when modifying normalizer parameters, ensuring rules are rebuilt.

from tn.chinese.normalizer import Normalizer as ZhNormalizer from itn.chinese.inverse_normalizer import InverseNormalizer from tn.english.normalizer import Normalizer as EnNormalizer # Chinese Text Normalization with erhua removal zh_tn_model = ZhNormalizer(remove_erhua=True, overwrite_cache=True) zh_tn_text = "你好WeTextProcessing 1.0，全新版本儿，简直666" print(f"Chinese TN: {zh_tn_text} => {zh_tn_model.normalize(zh_tn_text)}") # Chinese Inverse Text Normalization zh_itn_model = InverseNormalizer(enable_0_to_9=False, overwrite_cache=True) zh_itn_text = "你好WeTextProcessing 一点零，全新版本儿，简直六六六" print(f"Chinese ITN: {zh_itn_text} => {zh_itn_model.normalize(zh_itn_text)}") # English Text Normalization en_tn_model = EnNormalizer(overwrite_cache=True) en_tn_text = "Hello WeTextProcessing 1.0, life is short, just use wetext, 666, 9 and 10" print(f"English TN: {en_tn_text} => {en_tn_model.normalize(en_tn_text)}")

Debug

Known issues

breakingVersion 1.0.0 introduced significant changes to English Text Normalization rules, simplifying them compared to NeMo. While resulting in smaller FST sizes and faster build times, existing English TN implementations might require review and adjustment.

fix

Review English TN usage with version 1.0.0 or later. Test thoroughly to ensure desired normalization behavior. If migrating from older versions, be aware of potential changes in output for English text.

affects: >=1.0.0

gotchaThe `pynini` dependency, fundamental to WeTextProcessing, is primarily designed for Linux and macOS environments. Direct installation on Windows is not straightforward and often leads to errors. While there's a separate `wetext` package that doesn't depend on Pynini, `WeTextProcessing` requires it.

fix

For Windows users, it is highly recommended to use Windows Subsystem for Linux (WSL) or a Linux virtual machine for development and deployment. Alternatively, ensure you have a compatible `pynini` wheel for your specific Python version and platform before installing `WeTextProcessing`.

affects: All versions

gotchaIf you modify any parameters when initializing a `Normalizer` or `InverseNormalizer` (e.g., `remove_erhua`, `enable_0_to_9`), you must set `overwrite_cache=True` for the changes to take effect and for the underlying FSTs to be rebuilt. Failing to do so will result in the model reusing cached rules, ignoring your parameter changes.

fix

Always pass `overwrite_cache=True` in the constructor of `Normalizer` or `InverseNormalizer` if you are changing its parameters from the default, especially during initial setup or rule modification. For subsequent uses with the same parameters, `overwrite_cache=False` (the default) can be used to load compiled rules faster.

affects: All versions

gotchaStarting from version 1.0.1, the global logging configuration was disabled within the library to prevent it from overwriting logging levels of other programs in the same environment. If your application relies on WeTextProcessing configuring logging globally, this behavior has changed.

fix

If your application requires specific logging behavior from WeTextProcessing, configure logging explicitly in your application code rather than relying on the library's default global setup.

affects: >=1.0.1

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'tn'

Attempting to import `Normalizer` or `InverseNormalizer` directly from the top-level `wetextprocessing` package instead of the specific `tn.<lang>` or `itn.<lang>` submodules.

fix

Use explicit, language-specific imports. For example, `from tn.chinese.normalizer import Normalizer` for Chinese Text Normalization, or `from tn.english.normalizer import Normalizer as EnNormalizer` for English Text Normalization.

Failed to build wheel for pynini / ERROR: Could not build wheels for pynini which use PEP 517 and cannot be installed directly

The `pynini` dependency requires specific compilation steps and is primarily supported on Linux and macOS. This error typically occurs on Windows or other unsupported platforms during `pip install WeTextProcessing`.

fix

Install WeTextProcessing within a Linux environment (e.g., WSL on Windows) or ensure a pre-compiled `pynini` wheel compatible with your system and Python version is available and installed before installing WeTextProcessing. `conda install -c conda-forge pynini` is often recommended for Conda users.

TypeError: Normalizer() got an unexpected keyword argument 'remove_erhua'

You are likely trying to pass a language-specific parameter (like `remove_erhua` for Chinese) to a `Normalizer` instance that does not support it (e.g., an English normalizer, or a generic `wetext` normalizer if that package was used).

fix

Ensure you are using the correct language-specific Normalizer. For `remove_erhua`, you must use `from tn.chinese.normalizer import Normalizer`. For English, there are different or fewer configurable options. Consult the documentation for available parameters for each language's normalizer.

Upgrade

Version history

1.2.0latest on PyPI · released Jun 10, 2026

Audit

Dependencies

pyninirequiredCore dependency for building and running Finite State Transducers. Requires specific platform support (Linux/macOS).

importlib-resourcesrequiredUsed for resource loading, a common Python dependency.

Agent activity

94 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/wenet-e2e/WeTextProcessing ↗

packagehttps://pypi.org/project/wetextprocessing/ ↗