sparkdantic

library2.8.4pypypi✓ verified 22d ago

Sparkdantic is a Python library that bridges Pydantic models with PySpark schemas. It allows developers to define data structures using Pydantic, then automatically generate equivalent `pyspark.sql.types.StructType` schemas for use in Spark DataFrames. This simplifies data validation and schema management across Python applications and Spark environments. The current version is 2.8.0, and it maintains an active release cadence, frequently updating for Pydantic and PySpark compatibility.

data serialization database ai-ml

pip install sparkdantic pyspark

package ↗

INSTALL

IMPORT

SIG · SPARKDANTIC

sparkdantic

datapythonv2.8.4

Install

32.7s avg

Import

610ms

Disk

511MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v2.8.4 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 0.124s · 514.4MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 32.7s · import 0.120s · 514MB

511MB installed

● package 511MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

create_spark_schema

✓ from sparkdantic import create_spark_schema

BaseModel

✓ from pydantic import BaseModel

Pydantic's BaseModel is used to define the source models for schema generation.

This quickstart demonstrates how to define a Pydantic model and use `create_spark_schema` to generate a corresponding PySpark `StructType`. It then shows how to initialize a SparkSession and create an empty DataFrame using the generated schema, printing both the raw schema and the DataFrame's schema for verification.

from typing import Optional, List from pydantic import BaseModel from sparkdantic import create_spark_schema from pyspark.sql import SparkSession # Define your Pydantic model class Product(BaseModel): product_id: int name: str price: float description: Optional[str] = None tags: List[str] = [] # Generate the Spark schema from the Pydantic model spark_schema = create_spark_schema(Product) # Print the generated Spark schema (useful for verification) print("Generated Spark Schema:") print(spark_schema) # Optionally, use it with a Spark DataFrame spark = SparkSession.builder.appName("SparkdanticExample").getOrCreate() # Create an empty DataFrame with the defined schema df = spark.createDataFrame([], schema=spark_schema) print("\nDataFrame created with generated schema:") df.printSchema() spark.stop()

Debug

Known issues

breakingOlder versions of sparkdantic (< 2.0.0) do not support Pydantic V2. If you are upgrading your Pydantic dependency to V2, ensure you also upgrade sparkdantic to version 2.0.0 or higher.

fix

Upgrade sparkdantic to 2.0.0 or newer (e.g., `pip install sparkdantic>=2.0.0`). The current version 2.8.0 fully supports Pydantic V1 and V2.

affects: <2.0.0

gotchaPydantic `Enum` types are automatically mapped to `StringType` in Spark schemas. If you require more specific type handling or validation for enums within Spark, you might need to implement custom logic post-schema generation.

fix

Be aware of this automatic mapping. For stricter validation in Spark, consider UDFs or explicit casting after DataFrame creation based on the string values.

affects: All versions

gotchaPydantic `uuid.UUID` types are automatically converted to `StringType` in the generated Spark schema. Spark does not have a native UUID type, so string representation is the default and most common approach.

fix

Store UUIDs as strings in Spark DataFrames. If direct UUID object handling is needed in Spark, it typically involves string parsing within UDFs.

affects: All versions

Upgrade

Version history

2.8.4latest on PyPI · released Jul 13, 2026

Audit

Dependencies

pydanticrequiredCore dependency for defining data models. Supports Pydantic v1 and v2.

pysparkrequiredRequired for Spark schema generation and integration with Spark DataFrames.

Agent activity

13 hits · last 30 days

node

OpenAI (training)

Resources

packagehttps://pypi.org/project/sparkdantic/ ↗