Install & Compatibility
Where this runs
tested against v0.4.8 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.95 runs
installs and imports cleanly · install 0.0s · import 0.236s · 90.4MB
glibcpy 3.10–3.95 runs
installs and imports cleanly · install 3.8s · import 0.244s · 87MB
91MB installed
● package 91MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
msgpack_numpy
✓ import msgpack_numpy as m
Common alias for convenience.
patch
✓ m.patch()
Applies NumPy-aware encoding/decoding globally to msgpack functions.
encode
✓ msgpack_numpy.encode
Manual encoder for NumPy types.
decode
✓ msgpack_numpy.decode
Manual decoder for NumPy types.
The quickest way to use msgpack-numpy is to call `m.patch()` after importing `msgpack` and `msgpack_numpy`. This automatically configures `msgpack.packb` and `msgpack.unpackb` to handle NumPy data types. It's crucial to use `use_bin_type=True` during packing for binary data and `raw=False` (default) for unpacking strings.
import msgpack
import msgpack_numpy as m
import numpy as np
# Easiest way: Monkey-patch msgpack to be numpy-aware
m.patch()
# Example NumPy array
data = {'array': np.array([1.2, 3.4, 5.6], dtype=np.float32), 'scalar': np.float64(7.8)}
# Serialize
packed_data = msgpack.packb(data, use_bin_type=True)
print(f"Packed size: {len(packed_data)} bytes")
# Deserialize
unpacked_data = msgpack.unpackb(packed_data, raw=False)
# Verify
print(f"Original array type: {type(data['array'])}, dtype: {data['array'].dtype}")
print(f"Unpacked array type: {type(unpacked_data[b'array'])}, dtype: {unpacked_data[b'array'].dtype}")
print(f"Original scalar type: {type(data['scalar'])}, value: {data['scalar']}")
print(f"Unpacked scalar type: {type(unpacked_data[b'scalar'])}, value: {unpacked_data[b'scalar']}")
# Ensure unpacked arrays are modifiable if needed (they are read-only by default)
original_array = unpacked_data[b'array'].copy()
original_array[0] = 99.9
print(f"Modified array: {original_array}")
Debug
Known issues
breakingWhen upgrading the underlying `msgpack` library from versions `0.4` or earlier to `0.5` or later, the package name on PyPI changed from `msgpack-python` to `msgpack`. Users must `pip uninstall msgpack-python` before `pip install -U msgpack` to prevent conflicts. This directly affects `msgpack-numpy` installations.fixEnsure `msgpack-python` is uninstalled before installing or upgrading `msgpack`.
affects: msgpack < 0.5 transitioning to >= 0.5
gotchaNumPy arrays deserialized by `msgpack-numpy` are read-only views of the underlying data buffer to optimize memory usage. Attempting to modify them directly will result in an error.fixIf modifications are needed, create a writable copy using `.copy()`: `my_array = unpacked_data[b'my_array'].copy()`.
affects: All versions
gotchaThe primary design goal of `msgpack-numpy` is the preservation of numerical data types, which inherently adds some storage overhead to the serialized data (e.g., storing dtype, shape, kind).fixIf type preservation is not critical and absolute minimal size is, consider implementing a custom encoder/decoder for specific use cases or alternative serialization formats that explicitly sacrifice type information for size.
affects: All versions
gotcha`msgpack` (and by extension `msgpack-numpy`) has limits on object sizes. For instance, the maximum length of a binary object is `(2^32)-1` bytes (approximately 4GB). Attempting to serialize NumPy arrays significantly larger than this limit can fail.fixFor extremely large arrays (e.g., 10GB+), consider alternative serialization methods like `pickle`, HDF5, Zarr, or breaking the data into smaller chunks if `msgpack` is strictly required.
affects: All versions
gotchaNumPy arrays with `dtype='O'` (object type) are serialized/deserialized using Python's `pickle` module as a fallback within `msgpack-numpy`. This negates the efficiency benefits of `msgpack` and can introduce security risks if deserializing untrusted data.fixAvoid using NumPy object arrays with `msgpack-numpy` if possible. If necessary, consider a custom encoder/decoder to handle the specific objects efficiently, or be aware of the `pickle` overhead and security implications.
affects: All versions
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'msgpack-numpy'
The `msgpack-numpy` package has not been installed in the active Python environment or is not accessible.
fixInstall the package using pip: `pip install msgpack-numpy`
TypeError: buffer is too small for requested array
This error often occurs during deserialization when the buffer provided for a NumPy array does not match the expected size for the requested data type and shape, commonly with structured arrays or on platforms like Windows where default integer sizes might differ, or when dealing with multi-dimensional arrays where memory layout is miscalculated during unpacking.
fixEnsure that the data being deserialized was packed correctly and that the `msgpack-numpy.patch()` function or `encode`/`decode` hooks are applied consistently during both serialization and deserialization. For structured arrays or mixed dtypes, explicitly define precise dtypes during creation and ensure compatibility across platforms and versions. Reinstalling `msgpack-python`, `numpy`, and `msgpack-numpy` can sometimes resolve inconsistencies.
TypeError: Object of type ndarray is not JSON serializable
This error occurs when attempting to serialize a NumPy `ndarray` object (or other NumPy specific types like `numpy.float64`) using Python's standard `json` module, which does not inherently know how to convert NumPy types into JSON-compatible formats.
fixUse `msgpack-numpy` for serialization instead of `json`, or convert NumPy arrays to standard Python lists using `.tolist()` before JSON serialization. If using `msgpack-numpy`, ensure `msgpack_numpy.patch()` is called or `default=msgpack_numpy.encode` and `object_hook=msgpack_numpy.decode` are passed to `msgpack.packb` and `msgpack.unpackb` respectively.
TypeError: can't serialize X
This general `TypeError` indicates that `msgpack` (or `msgpack-numpy` if not correctly configured or if the type is truly unsupported) encountered an object it doesn't have a default encoder for, often seen with specific NumPy scalar types (e.g., `np.float32`) or complex Python objects that are not handled by `msgpack-numpy`'s default encoders.
fixEnsure `msgpack_numpy.patch()` is called before `msgpack` operations, or explicitly pass `default=msgpack_numpy.encode` to `msgpack.packb()` and `object_hook=msgpack_numpy.decode` to `msgpack.unpackb()`. If the specific type 'X' is a complex custom object or an unsupported NumPy `dtype='O'` array, consider implementing a custom encoder/decoder for it or converting it to a basic serializable type.
Upgrade
Version history
0.4.8latest on PyPI · released Jun 9, 2022
Audit
Dependencies
msgpackrequiredCore serialization library.
numpyrequiredProvides the array and numerical types to be serialized.