Registry / serialization / unicode-segmentation-rs

unicode-segmentation-rs

library0.2.4pypypi✓ verified 86d ago

unicode-segmentation-rs provides Python bindings for the Rust `unicode-segmentation` and `unicode-width` crates, offering robust Unicode text segmentation (grapheme clusters, words, sentences) and display width calculation according to Unicode standards. It is currently at version 0.2.4 and is an actively maintained library, with updates often tied to new Unicode standard releases in its underlying Rust dependencies.

serialization

pip install unicode-segmentation-rs

github ↗package ↗homepage ↗docs ↗

INSTALL

IMPORT

SIG · UNICODE-SEGMENTATI

unicode-segmentation-rs

serializationpythonv0.2.4

Install

1.5s avg

Import

—

Disk

16MB

Pass rate

8/ 10

Env Coverage8 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.2.4 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

glibc

py 3.10

✓ —

✓ 1.58s

py 3.11

✓ —

✓ 1.6s

py 3.12

✓ —

✓ 1.45s

py 3.13

✓ —

✓ 1.5s

py 3.9

✕ build_error

16MB installed

● package 16MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

graphemes

✓ from unicode_segmentation_rs import graphemes

unicode_words

✓ from unicode_segmentation_rs import unicode_words

unicode_sentences

✓ from unicode_segmentation_rs import unicode_sentences

text_width

✓ from unicode_segmentation_rs import text_width

gettext_wrap

✓ from unicode_segmentation_rs import gettext_wrap

Demonstrates basic usage for grapheme, word, and sentence segmentation, as well as calculating the display width of a string.

import unicode_segmentation_rs text = "Hello 👨‍👩‍👧‍👦 World. How are you?" # Grapheme clusters (user-perceived characters) graphemes = unicode_segmentation_rs.graphemes(text, is_extended=True) print(f"Graphemes: {graphemes}") # Unicode words (excludes punctuation and whitespace) words = unicode_segmentation_rs.unicode_words(text) print(f"Words: {words}") # Sentences sentences = unicode_segmentation_rs.unicode_sentences(text) print(f"Sentences: {sentences}") # Display width width = unicode_segmentation_rs.text_width("你好, World!") print(f"Display width of '你好, World!': {width}")

Debug

Known issues

gotchaWhen using `graphemes()` or `grapheme_indices()`, it is highly recommended to set `is_extended=True`. This adheres to the Unicode Standard Annex #29 for 'extended grapheme clusters', which represents user-perceived characters. Failing to do so can lead to non-intuitive or incorrect segmentation for complex Unicode sequences, such as emojis or combining characters.

fix

Always pass `is_extended=True` to `graphemes()` and `grapheme_indices()` unless you specifically require legacy grapheme clustering behavior.

affects: All versions

gotchaDirectly indexing Python strings after performing Unicode segmentation (e.g., trying to access `my_string[i]` based on grapheme cluster counts) is an anti-pattern. Unicode text segmentation algorithms are inherently streaming, and direct indexing into a string by a 'grapheme index' is inefficient and often indicative of a misunderstanding of Unicode text model. The library provides lists of segmented strings or indices, which should be iterated over, not used for direct string indexing.

fix

Iterate over the results of segmentation functions (e.g., `for grapheme in unicode_segmentation_rs.graphemes(text):`) rather than attempting direct string indexing with arbitrary offsets.

affects: All versions

gotchaWhile the Python bindings aim for stability, the underlying Rust `unicode-segmentation` crate has historically encountered panics (e.g., 'byte index is not a char boundary' or arithmetic overflows) with highly malformed or edge-case Unicode input, particularly with its lower-level cursor APIs. Although the Python layer should convert Rust panics into Python exceptions, unexpected input could still lead to issues or crashes in rare circumstances.

fix

Ensure input strings are valid Unicode where possible. For critical applications, implement robust error handling around calls to `unicode_segmentation_rs` functions, especially with untrusted input.

affects: All versions

Errors

Common errors & fixes

len(my_string) returning an incorrect 'character count'

Python's built-in `len()` counts Unicode code points, not user-perceived characters (grapheme clusters). This is a common misconception when dealing with complex scripts or emojis.

fix

`len(unicode_segmentation_rs.graphemes(my_string, is_extended=True))` will provide the count of user-perceived characters.

Crash or unexpected behavior when processing malformed UTF-8 input (e.g., `ValueError: byte index is not a char boundary`)

The underlying Rust library expects valid UTF-8 for its string operations. While Python strings are inherently Unicode-aware, constructing them from improperly decoded bytes or passing corrupted string data can lead to internal inconsistencies in the Rust layer.

fix

Always ensure that string inputs are correctly decoded from bytes if they originate from external sources (e.g., `my_bytes.decode('utf-8', errors='replace')`). The Python bindings should generally prevent direct exposure of raw byte-level issues unless the string itself is fundamentally corrupted.

Upgrade

Version history

0.2.4latest on PyPI · released Apr 10, 2026

Audit

Dependencies

No dependency data recorded yet.

Agent activity

9 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/WeblateOrg/unicode-segmentation-rs ↗

packagehttps://pypi.org/project/unicode-segmentation-rs/ ↗

homepagehttps://weblate.org/ ↗

docshttps://docs.weblate.org/ ↗