Registry / serialization / compress-pickle

compress-pickle

library2.1.0pypypi✓ verified 86d ago

compress-pickle is a Python library (version 2.1.0) that thinly wraps the standard `pickle` package with various standard compression libraries (gzip, bz2, lzma, zipfile, and optionally lz4). It provides an interface similar to `pickle.dump`, `pickle.load`, `pickle.dumps`, and `pickle.loads` to seamlessly serialize and deserialize Python objects to disk or file-like objects in a compressed manner. The library has an infrequent release cadence, with its last major update in September 2021.

serialization data

pip install compress-pickle

github ↗package ↗

INSTALL

IMPORT

SIG · COMPRESS-PICKLE

compress-pickle

serializationpythonv2.1.0

Install

1.8s avg

Import

66ms

Disk

22MB

Pass rate

5/ 10

Env Coverage5 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v2.1.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

glibc

py 3.10

4/8 runs

✓ 1.61s

py 3.11

4/8 runs

✓ 1.95s

py 3.12

4/8 runs

✓ 1.51s

py 3.13

4/8 runs

✓ 1.53s

py 3.9

4/8 runs

✓ 2.49s

22MB installed

● package 22MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

dump

✓ from compress_pickle import dump

Used for serializing and compressing an object to a file-like object.

load

✓ from compress_pickle import load

Used for decompressing and deserializing an object from a file-like object.

dumps

✓ from compress_pickle import dumps

Used for serializing and compressing an object to a bytes object.

loads

✓ from compress_pickle import loads

Used for decompressing and deserializing an object from a bytes object.

This quickstart demonstrates how to use `compress_pickle.dump` to serialize and compress a Python dictionary to a file, and `compress_pickle.load` to decompress and deserialize it back. The compression method (gzip in this case) is automatically inferred from the file extension '.pkl.gz'.

from compress_pickle import dump, load import os data = {'key': 'value', 'numbers': [1, 2, 3]} filename = 'my_compressed_data.pkl.gz' # Dump (serialize and compress) the data dump(data, filename) print(f"Data saved to {filename} with inferred gzip compression.") # Load (decompress and deserialize) the data loaded_data = load(filename) print(f"Data loaded successfully: {loaded_data}") # Clean up the created file os.remove(filename)

Debug

Known issues

breakingPython 3.5 support was dropped starting from version 2.0.0. If you need to support Python 3.5, you must install `compress-pickle==1.1.1`.

fix

Upgrade Python to 3.6+ or pin `compress-pickle` to version `1.1.1` for Python 3.5 environments.

affects: >=2.0.0

gotchaLike the underlying `pickle` module, `compress-pickle` is not secure against maliciously constructed data. Deserializing data from untrusted sources can lead to arbitrary code execution.

fix

Only unpickle data from trusted sources. Consider alternative serialization formats like JSON for untrusted data.

affects: All versions

gotchaFor very small Python objects, the overhead introduced by compression might cause the compressed file size to be larger than the uncompressed pickled data.

fix

Evaluate file sizes for your specific use case. Compression is generally beneficial for larger objects.

affects: All versions

gotchaBy default, `dump` and `load` infer the compression protocol from the file extension (e.g., '.pkl.gz' for gzip). If the file extension does not match the desired compression or is missing, explicitly set the `compression` argument (e.g., `compression='gzip'`).

fix

Always use appropriate file extensions or explicitly pass the `compression` argument to `dump` and `load`.

affects: All versions

gotchaThe `pickle` protocol used can affect compatibility. Objects pickled with newer Python versions or higher protocols may not be readable by older Python versions or lower protocols. `compress-pickle` inherits this behavior.

fix

Be mindful of the Python versions used for pickling and unpickling. Specify a compatible `pickler_kwargs={'protocol': N}` if cross-version compatibility is critical.

affects: All versions

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'compress_pickle'

The 'compress_pickle' library is not installed in your Python environment.

fix

Run `pip install compress-pickle` in your terminal to install the library.

_pickle.UnpicklingError: invalid load key

This error typically occurs when attempting to load a compressed pickle file using the wrong compression protocol or if the file is corrupted. The `compress-pickle` library expects the correct compression to be specified or inferred.

fix

Ensure you are loading the file with the same compression protocol used during saving. If the compression was inferred from the file extension, ensure the extension is correct. Example: `data = compress_pickle.load('file.pkl.gz', compression='gzip')`

OSError: Invalid data stream

This error indicates that the data being uncompressed is not in the expected format for the specified compression algorithm, often due to a mismatch between the compression used for saving and loading, or a corrupted file.

fix

Verify that the compression argument in `compress_pickle.load()` matches the compression method used when the file was saved. Also, check the integrity of the compressed file. Example: `data = compress_pickle.load('my_data.pkl.bz2', compression='bz2')`

object of type 'pickle.PickleBuffer' has no len()

This specific error can occur in `compress-pickle` when serializing large NumPy arrays with certain compression protocols other than 'gzip', particularly when a newer pickle protocol (like protocol 5) is implicitly used with an older `pickle` version or a specific compression backend that doesn't fully support `PickleBuffer` efficiently for all operations.

fix

Try using `compression='gzip'` for large NumPy arrays or explicitly specify an older pickle protocol (e.g., `protocol=4`) in `compress_pickle.dump` and `compress_pickle.load` to avoid issues with `PickleBuffer` handling. Example: `compress_pickle.dump(data, 'file.pkl.lz4', compression='lz4', protocol=4)`

Upgrade

Version history

2.1.0latest on PyPI · released Sep 21, 2021

Audit

Dependencies

lz4optionalProvides additional LZ4 compression support.

Agent activity

17 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/lucianopaz/compress_pickle ↗

packagehttps://pypi.org/project/compress-pickle/ ↗