Registry / llm-agents / openvino-tokenizers

openvino-tokenizers

library2026.2.1.0pypiunverified

OpenVINO Tokenizers provides utilities to convert pre-trained tokenizers, primarily from the Hugging Face `transformers` library, into OpenVINO models. These converted tokenizers can then be compiled and run efficiently on various hardware using the OpenVINO runtime, preparing text inputs for OpenVINO-optimized Large Language Models (LLMs). The current version is 2026.1.0.0, and releases typically align with major OpenVINO toolkit releases, often on a quarterly or yearly cadence.

llm-agents ai-ml

pip install openvino-tokenizers

INSTALL

IMPORT

SIG · OPENVINO-TOKENIZER

openvino-tokenizers

llm-agentsenv2026.2.1.0

Install

9.6s avg

Import

730ms

Disk

407MB

Pass rate

5/ 10

Env Coverage5 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v2026.2.1.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

glibc

py 3.10

2/4 runs

✓ 9.68s

py 3.11

2/4 runs

✓ 9.13s

py 3.12

2/4 runs

✓ 9.05s

py 3.13

2/4 runs

✓ 8.93s

py 3.9

2/4 runs

✓ 11.28s

407MB installed

● package 407MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

convert_tokenizer

✓ from openvino_tokenizers import convert_tokenizer

This quickstart demonstrates how to load a standard Hugging Face tokenizer and convert it into an OpenVINO model. The resulting `ov.Model` object can then be compiled and run by the OpenVINO runtime. Note that `transformers` is a common prerequisite for obtaining the initial tokenizer object.

from transformers import AutoTokenizer from openvino_tokenizers import convert_tokenizer import openvino as ov # 1. Load a Hugging Face tokenizer (requires `pip install transformers`) hf_tokenizer = AutoTokenizer.from_pretrained("gpt2") # 2. Convert the Hugging Face tokenizer to an OpenVINO model # The output is an ov.Model object ov_tokenizer_model = convert_tokenizer(hf_tokenizer, tokenizer_name="gpt2_ov_tokenizer") # 3. Print information about the converted OpenVINO model print(f"OpenVINO Tokenizer Model Name: {ov_tokenizer_model.get_friendly_name()}") print(f"Number of inputs: {len(ov_tokenizer_model.inputs)}") print(f"Number of outputs: {len(ov_tokenizer_model.outputs)}") # The `ov_tokenizer_model` can now be compiled and used with `openvino.Core()`

Debug

Known issues

gotchaThe `convert_tokenizer` function is primarily designed for tokenizers originating from the Hugging Face `transformers` library. Using custom tokenizer implementations or those from other NLP libraries may lead to conversion errors or unexpected behavior.

fix

Ensure the tokenizer object passed to `convert_tokenizer` is a standard Hugging Face tokenizer instance (e.g., from `AutoTokenizer.from_pretrained`). If using a custom tokenizer, verify it implements all necessary attributes and methods expected by OpenVINO Tokenizers.

affects: All versions

gotchaThe `openvino-tokenizers` package version is typically aligned with the major OpenVINO toolkit version it supports. Mismatching versions between `openvino-tokenizers` and the `openvino` runtime package can lead to compatibility issues, especially when dealing with advanced OpenVINO features or specific hardware support.

fix

Always install `openvino-tokenizers` and `openvino` packages with matching major version numbers (e.g., both 2024.2.x or both 2026.1.x) to ensure full compatibility.

affects: All versions

breakingChanges in OpenVINO's core operations or model representation between major OpenVINO toolkit versions (e.g., 2024.x to 2026.x) can lead to converted tokenizer models being incompatible with older or newer OpenVINO runtimes, or requiring code adjustments.

fix

Consult the release notes for both `openvino-tokenizers` and the main `openvino` toolkit when upgrading across major versions. Re-convert tokenizers with the new `openvino-tokenizers` version if you upgrade your `openvino` runtime.

affects: Across major OpenVINO versions (e.g., 2024.x -> 2026.x)

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'transformers'

The `transformers` library is not installed, but `AutoTokenizer` or other `transformers` components are being imported or used.

fix

Install the Hugging Face `transformers` library: `pip install transformers`.

RuntimeError: unsupported tokenizer type or missing attributes

The tokenizer object passed to `convert_tokenizer` is not a recognized type or lacks attributes/methods that OpenVINO Tokenizers expects for conversion (e.g., `vocab_size`, `ids_to_tokens`).

fix

Ensure you are using a standard Hugging Face tokenizer (e.g., from `AutoTokenizer.from_pretrained`). If using a custom tokenizer, verify it adheres to the interface expected by `openvino-tokenizers`, possibly by inspecting the source code for expected tokenizer properties.

TypeError: 'list' object cannot be interpreted as an integer

The `convert_tokenizer` function or the OpenVINO tokenizer model input expects a single string or a list of strings, but received an incorrect input format.

fix

Verify that the input to `convert_tokenizer` is a tokenizer object, and inputs to the compiled OpenVINO tokenizer model are in the expected format, typically a `str` or `List[str]`.

Upgrade

Version history

2026.2.1.0latest on PyPI · released Jun 17, 2026

Audit

Dependencies

transformersoptionalCommonly used to load pre-trained tokenizers which are then converted by openvino-tokenizers. Not a direct runtime dependency of openvino-tokenizers, but required for most typical workflows.

openvinorequiredCore OpenVINO library is required for compiling and running the converted tokenizer models.

Agent activity

25 hits · last 30 days

node

Amazon

OpenAI (training)

Resources

No resource links recorded.