Registry / data / pyspark-extension

pyspark-extension

library2.15.0.4.1pypypiunverified

A library providing useful extensions to Apache Spark, including DataFrame diff, column transformation utilities, Parquet metadata reading, Spark Connect support, and dependency installation helpers. Current version 2.15.0.4.1, supports Spark 3.2+, 4.0; release cadence is irregular with multiple releases per year.

data workflow

pip install pyspark-extension

github ↗package ↗

INSTALL

IMPORT

SIG · PYSPARK-EXTENSION

pyspark-extension

datapythonv2.15.0.4.1

Install

1.7s avg

Import

—

Disk

17MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v2.15.0.4.1 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.910 runs

installs and imports cleanly · install 0.0s · import 0.000s · 18.6MB

glibc

py 3.10–3.910 runs

installs and imports cleanly · install 1.7s · import 0.000s · 19MB

17MB installed

● package 17MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

diff

✓ from gresearch.spark.diff import diff

✗ from spark_extension import diff

ParquetMetadata

✓ from gresearch.spark.parquet import ParquetMetadata

ColumnTransformer

✓ from gresearch.spark.column import ColumnTransformer

Basic usage: diff two DataFrames

from pyspark.sql import SparkSession from spark_extension import diff spark = SparkSession.builder.appName("test").getOrCreate() df1 = spark.createDataFrame([(1, "a"), (2, "b")], ["id", "val"]) df2 = spark.createDataFrame([(1, "a"), (3, "c")], ["id", "val"]) result = diff(df1, df2) result.show()

Debug

Known issues

breakingRemoved support for Spark 3.0 and 3.1 starting from version 2.15.0. Upgrade to Spark 3.2+ or pin pyspark-extension <2.15.0.

fix

Use Spark 3.2+ or install pyspark-extension==2.14.2

affects: >=2.15.0

breakingAll undocumented unintended public API parts were made private in version 2.15.0. Any use of internal symbols (e.g., _internal methods) will break.

fix

Only use documented public API; check release notes for symbols that became private.

affects: >=2.15.0

gotchaThe Java/Scala jar is required for some features (e.g., diff on DataFrames with complex types). Install with pip install pyspark-extension[scala] or manually add the jar to Spark session.

fix

Install with [scala] extra or configure spark.jars

affects: all

deprecatedBackticks handling: In version 2.14.0, columns with special characters are quoted with backticks; columns with only alphanumerics and underscores are no longer quoted. This may break comparisons relying on quoted column names in SQL expressions.

fix

Review column name quoting in any custom SQL using column names from this library.

affects: >=2.14.0

Upgrade

Version history

2.15.0.4.1latest on PyPI · released Mar 18, 2026

Audit

Dependencies

pysparkrequiredCore dependency for Spark functionality

py4jrequiredUsed for Java bridge

Agent activity

38 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/G-Research/spark-extension ↗

packagehttps://pypi.org/project/pyspark-extension/ ↗