Install & Compatibility
Where this runs
No compatibility data collected yet for this library.
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
to_spark_schema
✓ from pydantic_spark import to_spark_schema
✗ from pydantic_spark import to_spark_schema
Basic usage: define a Pydantic model and convert to Spark schema.
from pyspark.sql import SparkSession
from pydantic import BaseModel
from pydantic_spark import to_spark_schema
class MyModel(BaseModel):
name: str
age: int
spark = SparkSession.builder.getOrCreate()
schema = to_spark_schema(MyModel)
df = spark.createDataFrame([], schema)
print(df.schema)
spark.stop()
Debug
Known issues
breakingVersion 1.0.0 dropped support for Pydantic v1. If upgrading from v0.3.0 or earlier, you must migrate your models to Pydantic v2.fixUpdate Pydantic to v2 and follow their migration guide (https://docs.pydantic.dev/latest/migration/)
affects: <1.0.0 to >=1.0.0
gotchaComplex nested types (e.g., models with Union, Optional, or recursive references) may produce unexpected Spark types. Manual schema adjustments might be needed.fixInspect the generated schema and override using pydantic Field(..., schema_extra={...}) or custom serialization. affects: all
deprecatedThe 'coerce' feature (CoerceType) from v0.3.0 is deprecated in v1.0.0+. Use Pydantic's built-in validators instead.fixRemove usage of CoerceType and replace with @field_validator or @model_validator in your Pydantic model.
affects: >=1.0.0
Upgrade
Version history
1.0.1latest on PyPI · released Nov 24, 2023
Audit
Dependencies
pydanticrequiredCore dependency for model definitions
pysparkoptionalTarget Spark environment