Install & Compatibility
Where this runs
tested against v1.5.11 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.95 runs
installs and imports cleanly · install 0.0s · import 0.126s · 20.6MB
glibcpy 3.10–3.95 runs
installs and imports cleanly · install 2.2s · import 0.108s · 22MB
19MB installed
● package 19MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
split_multi
✓ from segtok.segmenter import split_multi
web_tokenizer
✓ from segtok.tokenizer import web_tokenizer
split_contractions
✓ from segtok.tokenizer import split_contractions
word_tokenizer
✓ from segtok.tokenizer import word_tokenizer
This quickstart demonstrates basic sentence segmentation using `split_multi` and then tokenizes each sentence using `web_tokenizer` followed by `split_contractions` for English-specific handling.
from segtok.segmenter import split_multi
from segtok.tokenizer import web_tokenizer, split_contractions
text = "Hello, Mr. Man. He smiled!! This, i.e. that, is it. Don't worry."
sentences = split_multi(text)
all_tokens = []
for sentence in sentences:
tokens = list(split_contractions(web_tokenizer(sentence)))
all_tokens.append(tokens)
print("Original Text:", text)
print("\nSentences:")
for s in sentences:
print(f"- {s}")
print("\nTokens per sentence:")
for i, tokens in enumerate(all_tokens):
print(f"Sentence {i+1}: {tokens}")
Debug
Known issues
breakingThe `segtok` library is largely superseded by `syntok` (segtok v2), its direct successor. `syntok` offers better performance and fixes several tricky issues, particularly with sentence terminal markers not followed by spaces.fixConsider migrating to `syntok`. Install with `pip install syntok` and adjust imports and usage patterns. `syntok` is Python 3.6+ only.
affects: All segtok versions
gotchaOn Linux systems, installing the `regex` dependency (a core requirement for `segtok`) may fail if Python development headers (`python-dev` or `python3-dev`) are not installed.fixInstall the necessary development packages: `sudo apt-get install python3-dev` (Debian/Ubuntu) or `sudo yum install python3-devel` (CentOS/RHEL) before installing `segtok`.
affects: All versions on Linux
gotcha`segtok` is specifically designed and tuned for Indo-European languages (e.g., English, German, Spanish). Its performance and correctness may degrade significantly for other language families, such as CJK languages.fixFor non-Indo-European languages, evaluate alternatives designed for those specific linguistic characteristics. Do not assume `segtok` will perform well out-of-the-box.
affects: All versions
deprecatedWhile `segtok` itself works with Python 2.7 and 3.5+, its recommended successor, `syntok`, requires Python 3.6 or newer due to its reliance on the `typing` module.fixIf upgrading to `syntok` is desired, ensure your project runs on Python 3.6+.
affects: <=1.5.11
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'segtok'
The 'segtok' library has not been installed in the Python environment where the code is being run.
AttributeError: module 'segtok' has no attribute 'segment'
The 'segment' function is located within the 'segtok.segmenter' submodule and must be explicitly imported from there, rather than accessed directly via the top-level 'segtok' module.
fixfrom segtok.segmenter import segment
# Then use it like:
segment("Your text here.") ModuleNotFoundError: cannot import name 'tokenize' from 'segtok.segmenter'
The `segtok` library does not provide a `tokenize` function within its `segmenter` module; word tokenization is handled by `word_tokenize` in the `tokenizer` module, which is common confusion with the `syntok` library's API.
fixfrom segtok.tokenizer import word_tokenize
# Then use it like:
word_tokenize("Your text here.") TypeError: can only concatenate str (not list) to str
The `segment` (and `word_tokenize`) function expects a single string as input, but a non-string type (e.g., a list or other iterable) was provided.
fixsegment("This is a single string. Another sentence.") Upgrade
Version history
1.5.11latest on PyPI · released Dec 15, 2021
Audit
Dependencies
regexrequiredCore dependency for pattern-based segmentation and tokenization. On Linux, requires Python development headers (python-dev/python3-dev) for compilation.