sentencepiece

library0.2.2pypypi✓ verified 25d ago

SentencePiece is an unsupervised text tokenizer and detokenizer, primarily designed for Neural Network-based text generation systems where the vocabulary size is predetermined. It implements subword units like Byte-Pair Encoding (BPE) and Unigram Language Model, capable of training directly from raw sentences without pre-tokenization. The library is actively maintained with regular updates. The current version is 0.2.1.

ai-ml llm-agents

pip install sentencepiece

github ↗package ↗

INSTALL

IMPORT

SIG · SENTENCEPIECE

sentencepiece

ai-mlpythonv0.2.2

Install

1.7s avg

Import

60ms

Disk

19MB

Pass rate

5/ 10

Env Coverage5 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.2.2 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

build_error

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 1.7s · import 0.060s · 21MB

19MB installed

● package 19MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

SentencePieceProcessor

✓ import sentencepiece as spm sp = spm.SentencePieceProcessor()

SentencePieceTrainer

✓ import sentencepiece as spm spm.SentencePieceTrainer.train(...)

This quickstart demonstrates how to train a SentencePiece model from a text file, load the trained model, and then use it to encode text into subword pieces and IDs, and decode IDs back to text. The `input` parameter for training expects a file path.

import sentencepiece as spm import os # Create a dummy text file for training corpus_content = "This is a test sentence. SentencePiece is awesome.\nAnother example sentence for training." corpus_file = "corpus.txt" with open(corpus_file, "w", encoding="utf-8") as f: f.write(corpus_content) model_prefix = "m_model" vocab_size = 8000 # Train a SentencePiece model spm.SentencePieceTrainer.train( input=corpus_file, model_prefix=model_prefix, vocab_size=vocab_size ) # Load the trained model sp = spm.SentencePieceProcessor() sp.load(f"{model_prefix}.model") # Encode text text_to_encode = "SentencePiece tokenization is powerful." encoded_pieces = sp.encode_as_pieces(text_to_encode) encoded_ids = sp.encode_as_ids(text_to_encode) print(f"Original text: {text_to_encode}") print(f"Encoded pieces: {encoded_pieces}") print(f"Encoded IDs: {encoded_ids}") # Decode IDs back to text decoded_text = sp.decode_ids(encoded_ids) print(f"Decoded text: {decoded_text}") # Clean up generated model files os.remove(f"{model_prefix}.model") os.remove(f"{model_prefix}.vocab") os.remove(corpus_file)

spm_encode --version

Debug

Known issues

breakingStarting from version 0.2.0, `sentencepiece` requires Python 3.9 or newer. Users on older Python versions (e.g., 3.8 or below) will encounter installation failures.

fix

Upgrade your Python environment to 3.9 or later, or pin `sentencepiece` to `0.1.99` or an earlier compatible version.

affects: >=0.2.0

gotchaIf a pre-built wheel is not available for your specific Python version, operating system, or CPU architecture, `pip install sentencepiece` will attempt to build from source. This process requires a C++ compiler, CMake, and Python development headers to be installed on your system.

fix

Ensure that necessary build tools (like `cmake`, C++ compiler) and Python development headers are installed for your environment if `pip install` fails. It's often easier to use a Python version for which pre-built wheels are readily available.

affects: all

gotchaVersion 0.2.0 of `sentencepiece` had known compatibility issues with other libraries, specifically `transformers` and `tensorflow`, due to a flag redefinition that could lead to Python kernel crashes.

fix

Users encountering this issue with `v0.2.0` should upgrade to `v0.2.1` or a newer version, as the fix has been merged.

affects: 0.2.0

gotchaVersion 0.2.1 introduces experimental free-threading support. While `const` and `static` methods like `encode()` and `decode()` are designed to work without the GIL, non-const methods such as `load()` may have potential data race issues.

fix

If using `sentencepiece` in a free-threaded environment and calling non-const methods like `load()`, ensure appropriate explicit locks are implemented to prevent data races.

affects: >=0.2.1

gotchaWhen training a SentencePiece model, the `spm.SentencePieceTrainer.train()` method is optimized for file-based input, expecting a raw text file (typically one sentence per line). While it can accept an iterable, for large datasets, providing a file path (or a file-like object in environments with limited local filesystem access) is the standard and most efficient approach.

fix

Prepare your training data in a plain text file, with one sentence per line, and pass the file path to the `input` argument of `SentencePieceTrainer.train()`.

affects: all

breakingWhen training a SentencePiece model, if the input corpus is extremely small or has very limited unique characters/sequences, requesting a `vocab_size` that exceeds the maximum possible vocabulary derivable from the data can lead to a `RuntimeError` stating 'Vocabulary size too high' with a specific upper limit.

fix

Ensure your training corpus is sufficiently large and diverse to support the desired `vocab_size`. If the corpus is intentionally small, reduce the `vocab_size` parameter in `SentencePieceTrainer.train()` to a value less than or equal to the maximum allowed size specified in the error message (e.g., `<= 33` in this case).

affects: all

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'sentencepiece'

The `sentencepiece` Python package is not installed in the current environment or is not accessible in the Python path.

fix

Run `pip install sentencepiece` in your terminal to install the library.

sentencepiece.SentencePieceError: Cannot open file

The SentencePiece processor failed to load a model because the specified model file path is incorrect, the file does not exist, or the model file is corrupted.

fix

Verify that the `.model` file exists at the provided path, is accessible, and is a valid SentencePiece model file. Double-check the path spelling and file permissions.

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xXX in position Y: invalid start byte

The input text file being read by SentencePiece (e.g., for training or processing) is not encoded in UTF-8, but SentencePiece expects UTF-8 by default.

fix

Ensure your input text files are saved with UTF-8 encoding. If not possible, read the file with its correct encoding and then process the resulting string, or try specifying the encoding if the SentencePiece method supports it.

sentencepiece.SentencePieceError: Input file is empty.

The input file(s) provided to `spm.SentencePieceTrainer.train` for model training are empty, do not exist, or their paths are incorrect, resulting in no data for training.

fix

Confirm that the input file(s) specified in the `input` argument for `SentencePieceTrainer.train` contain valid text data and that their paths are correct and accessible.

Upgrade

Version history

0.2.2latest on PyPI · released Jul 12, 2026

Audit

Dependencies

No dependency data recorded yet.

Agent activity

20 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/google/sentencepiece ↗

packagehttps://pypi.org/project/sentencepiece/ ↗