Registry / ai-ml / curated-tokenizers

curated-tokenizers

library2.0.1pypypi✓ verified 23d ago

Curated Tokenizers is a lightweight Python library by Explosion (creators of spaCy) that provides efficient and production-ready implementations of various piece tokenization algorithms, including Byte-Pair Encoding (BPE), WordPiece, and SentencePiece. It focuses on fast, reliable tokenization suitable for integrating into larger NLP pipelines. The library is currently at version 2.0.0, with an active but less frequent release cadence focused on performance and stability.

ai-ml

pip install curated-tokenizers

github ↗package ↗

INSTALL

IMPORT

SIG · CURATED-TOKENIZERS

curated-tokenizers

ai-mlpythonv2.0.1

Install

2.4s avg

Import

119ms

Disk

40MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v2.0.1 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.910 runs

installs and imports cleanly · install 0.0s · import 0.126s · 41MB

glibc

py 3.10–3.910 runs

installs and imports cleanly · install 2.4s · import 0.111s · 43MB

40MB installed

● package 40MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

ByteBPEProcessor

✓ from curated_tokenizers import ByteBPEProcessor

WordPieceProcessor

✓ from curated_tokenizers import WordPieceProcessor

SentencePieceProcessor

✓ from curated_tokenizers import SentencePieceProcessor

✗ from curated_tokenizers.tokenizers import SentencePieceProcessor

All main processors are exposed directly at the top-level package for consistent access.

This quickstart demonstrates how to instantiate and use a `ByteBPEProcessor` for encoding and decoding text. Note that while this example creates a processor in memory, typical usage involves loading pre-trained models from files using `from_file` methods (e.g., for `vocab.json` and `merges.txt`).

from curated_tokenizers import ByteBPEProcessor # Create a minimal, in-memory ByteBPE processor for demonstration. # In a real application, you would load pre-trained models from files # using methods like `ByteBPEProcessor.from_file(vocab_path, merges_path)`. # Define a simple vocabulary mapping tokens (as bytes) to IDs token_to_id = { b"<unk>": 0, b"a": 1, b"b": 2, b"c": 3, b"ab": 4, b"abc": 5 } # Reverse mapping from IDs to tokens id_to_token = {v: k for k, v in token_to_id.items()} # Define some merge rules (as tuples of bytes) merges = [ (b"a", b"b"), (b"ab", b"c") ] # Instantiate the ByteBPEProcessor processor = ByteBPEProcessor( token_to_id=token_to_id, id_to_token=id_to_token, bpe_merges=merges, dropout=0.0, # Use 0.0 for deterministic tokenization unk_id=token_to_id[b"<unk>"] ) text = "abc abc" print(f"Original text: '{text}'") # Encode the text into a list of integer IDs encoded_ids = processor.encode(text) print(f"Encoded IDs: {encoded_ids}") # Decode the IDs back into a string decoded_text = processor.decode_from_ids(encoded_ids) print(f"Decoded text: '{decoded_text}'")

Debug

Known issues

breakingThe package was renamed from `cutlery` to `curated-tokenizers`. Users migrating from `cutlery` must update their import statements and package names.

fix

Change `pip install cutlery` to `pip install curated-tokenizers` and update all `from cutlery import ...` to `from curated_tokenizers import ...`.

affects: <0.0.7

gotchaThe `SentencePieceProcessor` requires the `sentencepiece` library, which is an optional dependency. It must be installed separately.

fix

Install with `pip install curated-tokenizers[sentencepiece]` if you plan to use `SentencePieceProcessor`.

affects: All

gotchaAll piece processors (ByteBPEProcessor, WordPieceProcessor, SentencePieceProcessor) are designed to load pre-trained models from files. Instantiating them directly in memory for simple demos (as done in the quickstart) is possible but often more complex than loading an existing model file.

fix

For production or realistic examples, use `Processor.from_file(path_to_model)` and ensure you have the necessary model files.

affects: All

gotchaVersion 2.0.0 primarily introduces performance improvements for Byte BPE encoding. While the API is generally stable, major version bumps can sometimes involve subtle behavioral shifts. Review your existing code for any unexpected changes.

fix

Thoroughly test existing tokenization logic after upgrading to v2.0.0 to ensure consistent output, especially for Byte BPE.

affects: >=2.0.0

Errors

Common errors & fixes

AttributeError: 'DocTransformerOutput' object has no attribute 'tensors'

This error typically arises from an incompatibility between the versions of `curated-transformers` and `curated-tokenizers` or an outdated `spaCy` version, where the `DocTransformerOutput` object's structure has changed and the expected `tensors` attribute is missing.

fix

Ensure all related packages (`curated-tokenizers`, `curated-transformers`, and `spaCy`) are updated to their latest compatible versions. If updating doesn't resolve it, a temporary fix might involve downgrading `curated-tokenizers` or `curated-transformers` to known compatible versions as indicated in project issue trackers.

ERROR: Could not build wheels for curated-tokenizers

This installation error occurs when pre-built binary wheels are not available for your specific Python version and operating system, requiring the package to be compiled from source. This compilation fails if necessary build tools (like a C++ compiler or Rust toolchain) are missing.

fix

Install the required build tools for your operating system (e.g., 'Build Tools for Visual Studio' on Windows, `build-essential` on Linux, Xcode Command Line Tools on macOS). Alternatively, try using a Python version for which pre-built wheels are readily available, or install `curated-transformers` as a dependency, which often handles `curated-tokenizers` installation.

ModuleNotFoundError: No module named 'curated_tokenizers'

This error means the `curated-tokenizers` library is either not installed, is installed in a different Python environment than the one currently active, or there is a typo in the import statement.

fix

Install the library using `pip install curated_tokenizers`. Verify that your Python environment is correctly activated and that the import statement, such as `import curated_tokenizers` or `from curated_tokenizers import ...`, is spelled correctly.

ValueError: [E002] Can't find factory for 'transformer' for language English (en)

This spaCy-related error occurs when attempting to add a `curated-transformers` component (which internally uses `curated-tokenizers`) to a spaCy pipeline, but the component factory is not correctly registered or named within the spaCy configuration. This often points to version mismatches or incorrect configuration.

fix

Ensure both `spaCy` and `spacy-curated-transformers` are installed and compatible. Verify the `factory` name for the transformer component in your spaCy configuration, typically `'curated-transformer'`. For new projects, use `python -m spacy init fill-curated-transformer` to generate a correctly configured pipeline.

Upgrade

Version history

2.0.1latest on PyPI · released Aug 24, 2026

Audit

Dependencies

regexrequiredRequired for some internal tokenization logic, implicitly installed.

sentencepieceoptionalRequired to use the SentencePieceProcessor. Must be installed as an optional extra.

Agent activity

24 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/explosion/curated-tokenizers ↗

packagehttps://pypi.org/project/curated-tokenizers/ ↗