Segments provides functions to tokenize and segment strings of text into individual characters or graphemes, and into segments according to orthography profiles. It is particularly useful for linguistic data processing using CLDF (Cross-Linguistic Data Formats). The library typically sees a few releases per year, with major versions introducing updates to Unicode standards.
pip install segmentsVerified import paths — ran on the pinned version, not inferred.
Demonstrates basic Unicode grapheme tokenization and custom segmentation using an orthography profile.
For rule-based segmentation, always instantiate `Segmenter` with a `profile` and use its `.segment()` method. Use `segments.tokenize()` exclusively for standard Unicode grapheme clustering.
Review tokenization outputs for critical linguistic data when upgrading from versions <2.0.0 to ensure consistency. Adjust downstream processing if specific grapheme clustering assumptions have changed due to updated Unicode interpretations.
Ensure the `profile` dictionary strictly adheres to the `{'rules': [[...]]}` structure. Refer to `pycldf` documentation for advanced orthography profile handling if loading from files or complex scenarios.