rustbpe

library0.1.0pypypi✓ verified 85d ago

RustBPE is a Python library that provides a fast Byte Pair Encoding (BPE) tokenizer implemented in Rust, with Python bindings. It is designed primarily for training GPT-style BPE tokenizers and offers features like parallel processing, GPT-4 style regex pre-tokenization, and direct export to the tiktoken format for efficient inference. Currently at version 0.1.0, it is an initial release, suggesting active and potentially rapid development.

ai-ml llm-agents

pip install rustbpe

package ↗

INSTALL

IMPORT

SIG · RUSTBPE

rustbpe

ai-mlpythonv0.1.0

Install

1.7s avg

Import

20ms

Disk

19MB

Pass rate

5/ 10

Env Coverage5 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.1.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.920 runs

build_error

glibc

py 3.10–3.920 runs

installs and imports cleanly · install 1.7s · import 0.020s · 21MB

19MB installed

● package 19MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

Tokenizer

✓ import rustbpe tokenizer = rustbpe.Tokenizer()

The primary class for BPE operations is directly available under the 'rustbpe' module.

This quickstart demonstrates how to initialize the `rustbpe.Tokenizer`, train it on a small dataset, and then use it to encode and decode text, including batch operations. It also shows the (optional) export capability to the tiktoken format.

import rustbpe import os # Create a tokenizer instance tokenizer = rustbpe.Tokenizer() # Prepare some sample training data training_texts = [ "hello world", "this is a test sentence", "rustbpe is fast and efficient" ] # Train the tokenizer # vocab_size is a crucial parameter defining the output vocabulary size tokenizer.train_from_iterator(training_texts, vocab_size=256) # Small vocab for example # Encode text text_to_encode = "hello rustbpe, how are you today?" ids = tokenizer.encode(text_to_encode) print(f"Encoded IDs: {ids}") # Decode IDs back to text decoded_text = tokenizer.decode(ids) print(f"Decoded Text: {decoded_text}") # Batch encode multiple texts (uses parallelization) batch_texts = ["text one", "text two", "text three"] all_ids = tokenizer.batch_encode(batch_texts) print(f"Batch Encoded IDs: {all_ids}") # Optional: Export to tiktoken format (requires tiktoken to be installed) # if os.environ.get('ENABLE_TIKTOKEN_EXPORT', 'false').lower() == 'true': # import tiktoken # tiktoken_tokenizer = tokenizer.export_to_tiktoken() # print("Tokenizer exported to tiktoken format.") print(f"Vocabulary size: {tokenizer.vocab_size}")

Debug

Known issues

gotchaRustBPE is optimized for training BPE tokenizers and exporting them to the tiktoken format for inference. While it offers encoding/decoding, its primary value proposition lies in the training aspect, distinguishing it from libraries solely focused on inference.

fix

Understand that rustbpe provides the 'missing tiktoken training code' and plan your workflow accordingly, potentially using tiktoken for production inference.

affects: 0.1.0+

gotchaThe library is in its initial `0.1.0` release. While tested, the API may be subject to changes and refinements in subsequent minor versions as the project matures.

fix

Pin your dependency to a specific minor version (`rustbpe==0.1.*`) in production environments and review release notes for breaking changes when updating.

affects: 0.1.0

gotchaRustBPE defaults to GPT-4 style regex pre-tokenization. If you need to match tokenization behavior of older GPT models (e.g., GPT-2/3) or other tokenizer types, this default pattern might yield different token splits.

fix

Review the tokenizer's pre-tokenization regex if exact compatibility with non-GPT-4 style models is critical. The library's source code indicates the specific pattern used.

affects: 0.1.0+

gotchaThe author notes that while the Python reference code is expert-level and equality tests pass, the underlying Rust implementation had significant AI assistance and might not be optimally structured by a Rust expert.

fix

While functional, be aware that future performance or maintainability improvements might arise from Rust-specific code optimizations. Report any observed inefficiencies or bugs on the GitHub issues page.

affects: 0.1.0+

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'rustbpe'

The `rustbpe` package is not installed in the current Python environment.

fix

Run `pip install rustbpe` to install the package.

TypeError: train_from_iterator() missing 1 required positional argument: 'vocab_size'

The `train_from_iterator` method requires `vocab_size` to be explicitly provided, which determines the final size of the tokenizer's vocabulary.

fix

Ensure `vocab_size` is passed as a keyword argument, e.g., `tokenizer.train_from_iterator(data_iterator, vocab_size=32768)`.

error: can't find Rust compiler

This error occurs when attempting to install `rustbpe` from source (e.g., if a pre-compiled wheel isn't available for your system or Python version) without a Rust toolchain installed.

fix

Install Rust by following instructions on `rustup.rs` (e.g., `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh`) and then retry `pip install rustbpe`. Ensure `maturin` is also installed if building directly from the git repository.

Upgrade

Version history

0.1.0latest on PyPI · released Jan 3, 2026

Audit

Dependencies

tiktokenoptionalCommonly used for inference after training a tokenizer with rustbpe, as rustbpe can export models to tiktoken's format.

Agent activity

45 hits · last 30 days

node

Amazon

OpenAI (training)

Resources

packagehttps://pypi.org/project/rustbpe/ ↗