Install & Compatibility
Where this runs
tested against v0.13.1 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.980 runs
installs and imports cleanly · install 0.1s · import 0.000s · 845.2MB
glibcpy 3.10–3.980 runs
installs and imports cleanly · install 42.0s · import 0.000s · 810MB
830MB installed
● package 830MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
make_reader
✓ from petastorm import make_reader
✗ from petastorm import make_reader
This quickstart demonstrates how to define a data schema, write sample data to a Parquet dataset using `make_writer`, and then read it back using `make_reader`. The example cleans up the temporary directory after execution. For real-world usage, consider configuring `reader_pool_type` and `num_epochs` based on your training requirements.
import os
import shutil
import numpy as np
from petastorm import make_reader, make_writer
from petastorm.unischema import Unischema, UnischemaField, ScalarCodec
from petastorm.codecs import CompressedNdarrayCodec
# 1. Define a schema for your data
MySchema = Unischema(
'MySchema',
[
UnischemaField('id', np.int32, (), ScalarCodec(np.int32), False),
UnischemaField('value', np.float64, (), ScalarCodec(np.float64), False),
UnischemaField('image', np.uint8, (10, 10, 3), CompressedNdarrayCodec(), False),
]
)
# 2. Define a dataset path (using a temporary local directory for example)
dataset_url = 'file:///tmp/petastorm_example_data'
# Clean up previous data if it exists
if os.path.exists('/tmp/petastorm_example_data'):
shutil.rmtree('/tmp/petastorm_example_data')
# 3. Write some dummy data to the Parquet dataset
print(f"Writing dummy data to {dataset_url}...")
with make_writer(dataset_url, MySchema, row_group_size_bytes=2 * 1024 * 1024) as writer:
for i in range(10):
writer.write(
MySchema.make_row(
id=i,
value=float(i * 10),
image=np.random.randint(0, 256, size=(10, 10, 3), dtype=np.uint8)
)
)
print(f"Successfully wrote 10 rows.")
# 4. Read data using make_reader
# reader_pool_type='thread' is often suitable for local development.
# For production, 'process' might be preferred depending on data access patterns.
print("\nReading data from the dataset:")
with make_reader(dataset_url, reader_pool_type='thread', num_epochs=1) as reader:
for i, row in enumerate(reader):
print(f"Row {i}: id={row.id}, value={row.value}, image_shape={row.image.shape}")
if i >= 2: # Print only a few rows for brevity
break
print("Finished reading example data.")
# Clean up the temporary dataset
shutil.rmtree('/tmp/petastorm_example_data')
Upgrade
Version history
0.13.1latest on PyPI · released Jan 2, 2026
Audit
Dependencies
numpyrequiredCore dependency for array handling.
pyarrowrequiredRequired for Parquet file I/O operations.
pandasoptionalOften used for data manipulation before writing to Parquet or after reading.
pysparkoptionalRequired for petastorm.spark module when integrating with Apache Spark.