tokie

library0.0.10pypypi✓ verified 81d ago

A blazingly fast tokenizer library that is 50x faster and 10x smaller than alternatives, providing 100% accurate encoding/decoding for large language models. Current version 0.0.10 targets Python >=3.10, with weekly release cadence.

ai-ml

pip install tokie

github ↗homepage ↗

INSTALL

IMPORT

SIG · TOKIE

tokie

ai-mlpythonv0.0.10

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

Tokenizer

✓ from tokie import Tokenizer

✗ from tokie import Tokie

Encoding

✓ from tokie import Encoding

tokie

✓ from tokie import tokie

Initialize a Tokie instance with a model name, then encode/decode strings.

from tokie import Tokie tok = Tokie(model='gpt2') encoded = tok.encode('Hello, world!') print(encoded) decoded = tok.decode(encoded) print(decoded)

Debug

Known issues

deprecatedThe `tokenizer` parameter renaming to `backend` in v0.1.0

fix

Use `backend='huggingface'` instead of `tokenizer='huggingface'`

affects: >=0.1.0

gotchaTokie must be installed with extra dependencies for non-HuggingFace backends (e.g., `pip install tokie[sentencepiece]`)

fix

Install with appropriate extras: `pip install tokie[all]` to cover all backends

affects: >=0.0.10

gotchaThe `model` parameter is required for initialization; no default model

fix

Provide a valid model name (e.g., 'gpt2', 'bert-base-uncased')

affects: all

Upgrade

Version history

0.0.10latest on PyPI · released May 28, 2026

Audit

Dependencies

tokenizersrequiredUnderlying tokenizer implementation for most models

Agent activity

28 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/chonkie-inc/tokie ↗

homepagehttps://github.com/chonkie-inc/tokie ↗