Registry / ai-ml / tensorflow-text

tensorflow-text

library2.21.0pypypi✓ verified 25d ago

TensorFlow Text is a library providing text-related operations, modules, and subgraphs for TensorFlow. It facilitates common text preprocessing tasks required by text-based models and offers features useful for sequence modeling not found in core TensorFlow. The library is actively maintained and typically releases new versions in lockstep with major and minor TensorFlow releases.

ai-ml

pip install tensorflow-text==2.20.1

github ↗package ↗

INSTALL

IMPORT

SIG · TENSORFLOW-TEXT

tensorflow-text

ai-mlpythonv2.21.0

Install

36.2s avg

Import

10986ms

Disk

2355MB

Pass rate

5/ 10

Env Coverage5 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v2.21.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.910 runs

build_error

glibc

py 3.10–3.910 runs

installs and imports cleanly · install 36.2s · import 10.986s · 2355.2MB

2355MB installed

● package 2355MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

tensorflow_text

✓ import tensorflow_text as tf_text

This quickstart demonstrates basic tokenization using the `WhitespaceTokenizer` from TensorFlow Text. It takes a TensorFlow string tensor and outputs a `RaggedTensor` of tokens, illustrating the common workflow for text processing within the TensorFlow graph.

import tensorflow as tf import tensorflow_text as tf_text # Create a WhitespaceTokenizer tokenizer = tf_text.WhitespaceTokenizer() # Input text as a TensorFlow tensor text_tensor = tf.constant(["Hello TensorFlow Text!", "This is a great library."]) # Tokenize the text tokens = tokenizer.tokenize(text_tensor) # Print the tokens (RaggedTensor output) print("Original text:", text_tensor.numpy()) print("Tokenized text:", tokens.numpy())

Debug

Known issues

breakingTensorFlow Text versions are tightly coupled with TensorFlow versions. Installing a `tensorflow-text` version that does not precisely match the minor version of your installed `tensorflow` can lead to import errors or runtime issues.

fix

Always install `tensorflow-text` with a minor version matching your `tensorflow` installation (e.g., `pip install tensorflow==2.20.0 tensorflow-text==2.20.1`). If upgrading TensorFlow, ensure `tensorflow-text` is upgraded concurrently to a compatible version.

affects: All versions

gotchaAfter TensorFlow Text version 2.10, pre-built pip packages are only provided for Linux x86_64 and Intel-based Macs. Users on other platforms (e.g., Windows, Aarch64, Apple Silicon Macs) may need to build from source.

fix

For unsupported platforms, consider building `tensorflow-text` from source, ensuring it's built in the same environment as your `tensorflow` installation, or using a supported platform.

affects: >=2.11.0

gotchaOlder versions of `FastWordpieceTokenizer` and `WhitespaceTokenizer` contained memory safety bugs (e.g., concerning `StringVocab` lifetime or out-of-bounds reads).

fix

Upgrade to TensorFlow Text 2.20.1 or later to ensure these memory safety fixes are applied.

affects: <2.20.1 (FastWordpieceTokenizer), <2.18.0 (WhitespaceTokenizer)

gotchaSome text operations in older versions had input size limitations (e.g., using `int16_t`), which could cause issues with large inputs.

fix

Upgrade to TensorFlow Text 2.20.0 or later, which updated input sizes to `int32_t` to support larger inputs.

affects: <2.20.0

gotchaPunctuation definition mismatches between different Unicode versions were observed in earlier releases, potentially leading to inconsistent tokenization.

fix

Upgrade to TensorFlow Text 2.19.0 or later, which includes fixes to handle these mismatches.

affects: <2.19.0

deprecatedThe `use_unique_shared_resource_name` option was removed in version 2.16.1. Code relying on this option will break.

fix

Remove any usage of `use_unique_shared_resource_name` from your code. Review the migration guides for TensorFlow Text 2.16.1 if this was explicitly used.

affects: >=2.16.1

Upgrade

Version history

2.21.0latest on PyPI · released Aug 20, 2026

Audit

Dependencies

tensorflowrequiredTensorFlow Text is built on TensorFlow and requires a tightly coupled version. The minor version of tensorflow-text must match the minor version of tensorflow (e.g., tensorflow-text==2.x.y requires tensorflow==2.x.*).

numpyrequiredCommon dependency for numerical operations in the TensorFlow ecosystem.

dm-treerequiredExplicitly limited to 0.1.8 in v2.19.0 release notes.

Agent activity

16 hits · last 30 days

node

OpenAI (training)

Resources

githubhttp://github.com/tensorflow/text ↗

packagehttps://pypi.org/project/tensorflow-text/ ↗