pydantic-to-pyarrow is a Python library (current version 0.1.6) designed to facilitate the conversion of Pydantic models into Apache PyArrow schemas. It streamlines data processing pipelines by allowing validation with Pydantic and subsequent conversion to a columnar format for efficient processing with PyArrow, Pandas, or Polars, and storage in formats like Parquet. The library is actively maintained with regular feature releases.
pip install pydantic-to-pyarrowVerified import paths — ran on the pinned version, not inferred.
This quickstart defines a nested Pydantic model (`Person` containing `Address`) with various field types including optional fields, lists, datetime, and UUID. It then uses `get_pyarrow_schema` to generate the corresponding PyArrow schema, demonstrating the library's primary functionality. The output shows how Pydantic types map to PyArrow types.
Ensure PyArrow version is 15.0 or higher, or explicitly pin NumPy to 1.x (e.g., `numpy<2`).
Manually verify integer ranges if concerned about overflows, or explicitly define smaller PyArrow integer types if appropriate.
Add a Pydantic serializer to your `UUID` field, e.g., `uuid_id: UUID = Field(json_schema_extra={'pyarrow_serializer': lambda uuid: uuid.bytes})`.To allow conversion with timezone loss, pass `allow_losing_tz=True` to `get_pyarrow_schema`. Example: `get_pyarrow_schema(MyModel, allow_losing_tz=True)`.
Upgrade `pyarrow` to version 15.0 or higher: `pip install --upgrade pyarrow`. Alternatively, if an upgrade is not possible, downgrade `numpy` to a 1.x version: `pip install "numpy<2"`.
Check the PyArrow documentation for supported Python versions. Consider using a slightly older, supported Python version, or wait for PyArrow to release wheels for your specific Python version. Sometimes, installing build dependencies (e.g., `pip install cython setuptools wheel`) can help, but a missing wheel for the specific Python version is usually the root cause.
Review the `pydantic-to-pyarrow` documentation or source for supported type conversions. If your type is not supported, consider transforming it to a compatible type within your Pydantic model (e.g., converting a custom object to a `str` or `dict`) or contributing support to the library. For Enums, ensure `pydantic-to-pyarrow` version is at least 0.1.2.