msgspec is a fast serialization and validation library, with builtin support for JSON, MessagePack, YAML, and TOML. It features high-performance encoders/decoders, zero-cost schema validation using Python type annotations, and a speedy `Struct` type. The library is actively maintained with frequent releases.
pip install msgspecVerified import paths — ran on the pinned version, not inferred.
Defines a simple `User` struct, encodes it to JSON, and then decodes it back. It also demonstrates how `msgspec` handles type validation errors during decoding. Uses `msgspec.field(default_factory=set)` for mutable default values to prevent common Python footguns.
Upgrade Python to 3.9+.
Review usage of `Encoder.encode_into` and ensure buffer management aligns with the new behavior, potentially pre-allocating larger buffers or handling re-allocation.
Replace calls to `from_builtins` with `msgspec.convert`.
Only use `memoryview` for zero-copy scenarios when strict performance is needed and memory lifecycle is precisely controlled. For most cases, prefer `bytes` or `bytearray` to ensure the input buffer can be garbage collected.
Always use `msgspec.field(default_factory=my_callable)` for mutable default values, where `my_callable` is a zero-argument function that returns a new mutable object (e.g., `list`, `set`, `dict`).
Cache the result of `msgspec.structs.fields()` if it's called repeatedly for the same `Struct` type, or redesign code to minimize its usage in hot paths.
Update `Struct` definitions and instantiation calls from `nogc=True` to `gc=False`.
For Python 3.9 and older, replace type hints like `email: str | None` with `email: typing.Optional[str]` or `email: typing.Union[str, None]`. Remember to import `typing` if not already done.
Ensure input data types precisely match the `msgspec.Struct` field annotations. If type coercion is required, implement custom hooks or preprocess the data before decoding.
Reorder the fields in your `msgspec.Struct` definition so that all required fields come before optional fields, or set `kw_only=True` in the `Struct` definition to make all fields keyword-only.
```python
import msgspec
# Fix 1: Reorder fields
class ValidOrder(msgspec.Struct):
a: str # Required
b: int = 0 # Optional
# Fix 2: Use kw_only=True
class KeywordOnly(msgspec.Struct, kw_only=True):
a: str = "" # Optional, but position doesn't matter for kw_only
b: int # Required
```Ensure that the input data matches the type annotations specified in your `msgspec.Struct` or the `type` argument passed to the decoder. Inspect the input JSON/MessagePack and the `msgspec.Struct` definition to find the mismatch.
```python
import msgspec
class User(msgspec.Struct):
name: str
groups: list[str] = msgspec.field(default_factory=list)
# Correct input: groups contains only strings
valid_data = b'{"name":"bob","groups":["devops"]}'
user = msgspec.json.decode(valid_data, type=User)
print(user)
# Original problematic input (assuming it contained an int where str was expected)
# invalid_data = b'{"name":"bob","groups":["devops", 123]}'
# try:
# msgspec.json.decode(invalid_data, type=User)
# except msgspec.ValidationError as e:
# print(e)
```Convert unsupported third-party types to a supported Python native type (e.g., `float`, `int`, `list`, `dict`) before encoding with `msgspec`. For custom types, you can also provide an `enc_hook` to the encoder.
```python
import msgspec
import numpy as np
# Original problematic code:
# mjson.encode(np.float64(1.0))
# Fix 1: Convert to a native Python float
value_np = np.float64(1.0)
encoded_data = msgspec.json.encode(float(value_np))
print(encoded_data)
# Fix 2: Use an enc_hook for custom handling
def numpy_encoder_hook(obj):
if isinstance(obj, np.ndarray):
return obj.tolist()
if isinstance(obj, np.generic):
return obj.item()
raise NotImplementedError
encoder = msgspec.json.Encoder(enc_hook=numpy_encoder_hook)
encoded_data_with_hook = encoder.encode(np.array([1.0, np.float64(2.0)]))
print(encoded_data_with_hook)
```If you need to modify the object, either define the `msgspec.Struct` without `frozen=True`, or create a new instance with the desired changes.
```python
import msgspec
class Point(msgspec.Struct, frozen=True):
x: float
y: float
p = Point(1.0, 2.0)
# Original problematic code:
# p.x = 2.0
# Fix: Create a new instance with updated values
p_new = Point(x=3.0, y=p.y) # Or use msgspec.structs.replace if you have many fields
print(p_new)
# If mutability is desired, define the struct without frozen=True
class MutablePoint(msgspec.Struct):
x: float
y: float
m_p = MutablePoint(1.0, 2.0)
m_p.x = 3.0
print(m_p)
```Refactor your type annotations to avoid unions with multiple string-like types. If you need to handle both `str` and `bytes`, consider using a single, unambiguous type or process `bytes` separately (e.g., base64 encode/decode if using JSON) or use `msgspec.Raw` for manual handling.
```python
import msgspec
from typing import Union
# Original problematic code:
# class TestData(msgspec.Struct):
# content: Union[str, bytes]
# Fix 1: Use a single, unambiguous type
class TestDataStr(msgspec.Struct):
content: str
class TestDataBytes(msgspec.Struct):
content: bytes
# Fix 2: If you must handle both, define separate structs and use a Tagged Union
# This allows msgspec to differentiate between them
class MyStrData(msgspec.Struct, tag='str_data'):
value: str
class MyBytesData(msgspec.Struct, tag='bytes_data'):
value: bytes
class Wrapper(msgspec.Struct):
data: Union[MyStrData, MyBytesData]
encoder = msgspec.msgpack.Encoder()
decoder = msgspec.msgpack.Decoder(Wrapper)
wrapped_str = Wrapper(MyStrData('hello'))
encoded_str = encoder.encode(wrapped_str)
decoded_str = decoder.decode(encoded_str)
print(decoded_str)
wrapped_bytes = Wrapper(MyBytesData(b'world'))
encoded_bytes = encoder.encode(wrapped_bytes)
decoded_bytes = decoder.decode(encoded_bytes)
print(decoded_bytes)
```