tokenizers

library0.23.1pypypi✓ verified 26d ago

Tokenizers is a Python library providing fast and versatile tokenization tools, optimized for both research and production environments. The current version is 0.22.2, released on January 5, 2026. The library is actively maintained with regular updates to enhance performance and add features.

llm-agents ai-ml

pip install tokenizers

github ↗package ↗

INSTALL

IMPORT

SIG · TOKENIZERS

tokenizers

llm-agentspythonv0.23.1

Install

4.9s avg

Import

25ms

Disk

68MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.23.1 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 0.026s · 75.6MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 4.9s · import 0.024s · 58MB

68MB installed

● package 68MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

Tokenizer

✓ from tokenizers import Tokenizer

Ensure correct import path to avoid ImportError

A simple example demonstrating how to load a pretrained tokenizer and tokenize a sample text.

from tokenizers import Tokenizer # Load a pretrained tokenizer tokenizer = Tokenizer.from_pretrained('bert-base-uncased') # Tokenize a text output = tokenizer.encode('Hello, world!') print(output.tokens)

Debug

Known issues

breakingPython 3.13 compatibility issues during installation

fix

Use Python 3.12 or earlier for installation; Python 3.13 is not supported due to PyO3 compatibility issues.

affects: 0.22.2

gotchaEnsure correct import path to avoid ImportError

fix

Use 'from tokenizers import Tokenizer' to import the Tokenizer class.

affects: All

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'tokenizers'

The 'tokenizers' Python package has not been installed in the current environment or the Python interpreter is not using the correct environment where it is installed.

fix

pip install tokenizers

Failed building wheel for tokenizers

The 'tokenizers' library contains Rust components that require a Rust compiler to be installed and accessible in your system's PATH during the pip installation process.

fix

Install Rust (e.g., using `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh`) and ensure it's in your system's PATH before retrying `pip install tokenizers`.

ValueError: Path /path/to/model is not a valid tokenizers model directory.

The path provided to `Tokenizer.from_file()` or `Tokenizer.from_pretrained()` does not point to a valid `tokenizer.json` file or a directory containing a valid tokenizer model.

fix

Ensure the specified path points directly to a `tokenizer.json` file or a directory recognized as a valid model by the `tokenizers` library, or use a correct Hugging Face model identifier.

TypeError: Expected a list of strings, received ...

The `encode_batch` method (or `encode` when used with multiple inputs) of the `tokenizers.Tokenizer` object received an input that was not a list of strings, or the list contained non-string elements.

fix

Ensure the input to `tokenizer.encode_batch()` is a list of strings, or for single input, ensure `tokenizer.encode()` receives a single string. Example: `tokenizer.encode_batch(['text1', 'text2'])`.

AttributeError: 'Encoding' object has no attribute 'input_ids'

When using the `tokenizers` library directly, the `encode()` method returns an `Encoding` object, which has different attribute names (e.g., `ids`, `attention_mask`, `type_ids`) compared to the `transformers.BatchEncoding` object.

fix

Access the token IDs via `encoding.ids`, attention mask via `encoding.attention_mask`, and token type IDs via `encoding.type_ids`. For example, `input_ids = encoding.ids`.

Upgrade

Version history

0.23.1latest on PyPI · released Apr 27, 2026

Audit

Dependencies

rustoptionalRequired for building from source; not needed for pip installation

Agent activity

36 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/huggingface/tokenizers ↗

packagehttps://pypi.org/project/tokenizers/ ↗