Install & Compatibility
Where this runs
tested against v4.1.1 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.95 runs
installs and imports cleanly · install 0.0s · import 0.264s · 18.4MB
glibcpy 3.10–3.95 runs
installs and imports cleanly · install 2.1s · import 0.228s · 19MB
19MB installed
● package 19MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
chunkerify
✓ from semchunk import chunkerify
✗ import semchunk
chunk
✓ from semchunk import chunk
✗ import semchunk
Chunker
✓ from semchunk import Chunker
✗ import semchunk
Demonstrates how to initialize a `semchunk` chunker with a specified tokenizer (e.g., by model name) and a maximum `chunk_size`, then use the returned callable chunker to split a given text into semantically meaningful segments. This example highlights the use of model names for easy tokenizer integration.
import semchunk
# You can optionally import transformers or tiktoken for specific tokenizers,
# but they are not direct dependencies of semchunk itself.
# from transformers import AutoTokenizer
# import tiktoken
chunk_size = 4 # A low chunk size is used here for demonstration purposes.
# Keep in mind, `semchunk` does not know how many special tokens, if any,
# your tokenizer adds to every input, so you may want to deduct the number
# of special tokens added from your chunk size.
text = 'The quick brown fox jumps over the lazy dog.'
# `chunkerify` accepts the name of an OpenAI model, Tiktoken encoding, Hugging Face model,
# or a custom tokenizer/token counter.
chunker = semchunk.chunkerify('gpt-4', chunk_size) # Using an OpenAI model name
# Example with a Hugging Face tokenizer (requires `transformers` to be installed):
# from transformers import AutoTokenizer
# chunker = semchunk.chunkerify(AutoTokenizer.from_pretrained('bert-base-uncased'), chunk_size)
chunks = chunker(text)
print(chunks)
# Expected output might vary slightly based on the tokenizer and chunk_size,
# but will be similar to: ['The quick brown', 'fox jumps over', 'the lazy dog.']
Debug
Known issues
gotchaWhen specifying `chunk_size`, be aware that `semchunk` does not automatically account for special tokens added by your tokenizer. You should typically deduct the number of special tokens from your desired `chunk_size` to ensure chunks do not exceed the model's actual context window. This critical guidance was removed in v3.0.0 but re-added in v3.1.1 due to its importance for correct usage.fixManually calculate and deduct special token count from your `chunk_size` parameter, or use the `tokenizer_kwargs` argument in `chunkerify()` (available since v4.0.0) to explicitly control tokenizer behavior regarding special tokens.
affects: >=3.1.1 (clarification), v3.0.0 (where it was missing)
breakingAs of version 4.0.0, all arguments to `semchunk.chunkerify()` and the callable `chunker` (returned by `chunkerify`) except for the first two/three arguments (respectively) are now keyword-only. Passing these arguments positionally will raise a `TypeError`.fixUpdate calls to `semchunk.chunkerify(...)` and the returned `chunker(...)` to use keyword arguments for all parameters after `tokenizer_or_token_counter` and `chunk_size` (for `chunkerify`), and after `text`, `chunk_size`, and `token_counter` (for the returned chunker).
affects: >=4.0.0
breakingIn version 4.0.0, `semchunk` changed its default behavior for handling special tokens when using `tiktoken` or `transformers` tokenizers. It now treats special tokens as normal text. Previously, `tiktoken` would raise an error, and `transformers` would treat them as special tokens. This can alter token counts and chunking behavior for texts containing special tokens.fixAdjust `chunk_size` expectations or utilize the new `tokenizer_kwargs` argument in `chunkerify()` to explicitly control how special tokens are handled by the underlying tokenizer if the new default behavior is not desired.
affects: >=4.0.0
gotchaVersion 3.2.0 introduced a significant improvement in chunk quality, particularly for low chunk sizes or documents with minimal whitespace, by prioritizing more semantically meaningful split points. Version 3.2.4 also fixed the splitter sorting order. While an improvement, these changes mean the exact chunk boundaries may differ from previous versions, which could impact downstream tasks sensitive to precise chunk content.fixReview your chunking outputs with `semchunk` versions 3.2.0 and later if your application relies on specific chunk boundaries, especially for texts with complex whitespace or when using low chunk sizes.
affects: >=3.2.0
Upgrade
Version history
4.1.1latest on PyPI · released Jun 13, 2026
Audit
Dependencies
isaacusoptionalRequired for AI-powered chunking mode and processing Isaacus Legal Graph Schema (ILGS) Documents.