Install & Compatibility
Where this runs
tested against v0.0.18.dev20250227 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
py 3.13
✕ build_error
✕ build_error
2995MB installed
● package 2995MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
Task
✓ import seqio
from seqio import Task
Mixture
✓ import seqio
from seqio import Mixture
FeatureConverter
✓ import seqio
from seqio import FeatureConverter
get_dataset
✓ import seqio
seqio.get_dataset(...)
FunctionDataSource
✓ from seqio.dataset_providers import FunctionDataSource
PassThroughVocabulary
✓ from seqio.vocabularies import PassThroughVocabulary
preprocessors
✓ from seqio import preprocessors
This quickstart defines a simple SeqIO Task with a `FunctionDataSource` that yields two text examples. It then uses `seqio.get_dataset` to retrieve and print the first three processed examples. This demonstrates the basic steps of defining data sources, tasks, and obtaining a `tf.data.Dataset`.
import seqio
from seqio.dataset_providers import FunctionDataSource
from seqio.vocabularies import PassThroughVocabulary
import tensorflow as tf
def my_text_generator():
yield {'text': 'hello world'}
yield {'text': 'seqio example'}
# Define a data source
my_data_source = FunctionDataSource(
dataset_fn=lambda split, shuffle: tf.data.Dataset.from_generator(
my_text_generator,
output_signature={
'text': tf.TensorSpec(shape=(), dtype=tf.string)}
),
splits=["train"],
caching_permitted=False
)
# Register a task (or define it directly)
seqio.Task.make_module(
"my_simple_task",
source=my_data_source,
preprocessors=[], # No preprocessing for simplicity
output_features={
"text": seqio.Feature(vocabulary=PassThroughVocabulary(), add_eos=False)
},
metric_fns=[]
)
# Get the dataset
dataset = seqio.get_dataset(
task_or_mixture_name="my_simple_task",
split="train",
sequence_length={
"text": 32 # Example sequence length
},
shuffle=False,
seed=0
)
print("First 3 examples from the dataset:")
for i, example in enumerate(dataset.take(3)):
print(f"Example {i}: {example['text'].numpy().decode('utf-8')}")
Debug
Known issues
breakingAs a nightly release, `seqio-nightly` is on the bleeding edge of development and may introduce frequent API changes or instabilities without deprecation periods. It is not recommended for production use.fixRefer to the latest GitHub repository for up-to-date API usage. Consider using the stable `seqio` release for more stability.
affects: All nightly versions
deprecated`seqio` is a refactor of the `t5.data` library. Users migrating from `t5.data` may encounter API differences.fixConsult the `seqio` documentation and migration guides on the GitHub repository to understand the updated API patterns for tasks, mixtures, and data sources.
affects: All versions (migration from t5.data)
gotchaWhen defining `seqio.Feature` or other classes, mutable default arguments (e.g., lists, dictionaries, or vocabulary objects directly) can lead to unexpected shared state bugs.fixAlways use `default_factory=lambda: ...` for mutable defaults in class definitions, or ensure new instances are created for each feature definition. (e.g., `vocabulary=PassThroughVocabulary()` vs `vocabulary=PassThroughVocabulary` if it were mutable in context).
affects: All versions
gotchaUsing `FunctionDataSource` with a `dataset_fn` that incorrectly handles `shuffle` or positional arguments can lead to unexpected data behavior or errors.fixEnsure the `dataset_fn` provided to `FunctionDataSource` correctly accepts and utilizes the `split` and `shuffle` arguments, returning a `tf.data.Dataset` with the expected output signature. Refer to examples for correct `output_signature` definition.
affects: All versions
Errors
Common errors & fixes
ValueError: mutable default <class 'seqio.vocabularies.PassThroughVocabulary'> for field vocabulary is not allowed: use default_factory
Attempting to use a mutable object (like an instance of `PassThroughVocabulary`) as a default value for a field in a class definition directly, which can lead to shared state across instances.
fixInstead of `vocabulary=PassThroughVocabulary()`, use `vocabulary=seqio.Feature(vocabulary=lambda: PassThroughVocabulary(), ...)` or define the default within the `output_features` dictionary as shown in the quickstart, ensuring a fresh instance is created each time.
TypeError: dataset_fn() got an unexpected keyword argument 'shuffle'
The `dataset_fn` provided to `FunctionDataSource` does not correctly define its signature to accept the `shuffle` argument (and potentially `split`).
fixModify your `dataset_fn` to accept `split` and `shuffle` as arguments, e.g., `dataset_fn=lambda split, shuffle: ...`. If `shuffle` is not used, it should still be accepted.
Upgrade
Version history
0.0.18.dev20250227latest on PyPI · released Feb 27, 2025
Audit
Dependencies
tensorflowrequiredCore data pipelines are built on `tf.data.Dataset`.
numpyrequiredFor converting `tf.data.Dataset` to NumPy iterators for JAX/PyTorch compatibility.