Registry / testing / pyspark-regression

pyspark-regression

library4.2.4pypypi✓ verified 86d ago

PySpark Regression is a Python library for regression testing Spark DataFrames, enabling comparison of outputs between different runs or environments. Version 4.2.4 is current; release cadence is irregular but active.

testing data

pip install pyspark-regression

github ↗package ↗docs ↗

INSTALL

IMPORT

SIG · PYSPARK-REGRESSION

pyspark-regression

testingpythonv4.2.4

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

RegressionTest

✓ from pyspark_regression import RegressionTest

✗ from pyspark_regression.testing import RegressionTest

Common mistake: assuming RegressionTest is in a submodule; it's directly in the top-level package.

Basic usage: compare two Spark DataFrames with a key column.

from pyspark.sql import SparkSession from pyspark_regression import RegressionTest spark = SparkSession.builder.appName('test').getOrCreate() df1 = spark.createDataFrame([(1, 'a'), (2, 'b')], ['id', 'val']) df2 = spark.createDataFrame([(1, 'a'), (2, 'b')], ['id', 'val']) tester = RegressionTest() result = tester.compare(df1, df2, key='id') print(result) # Should indicate equality

Debug

Known issues

breakingIn version 2.x, the API used `compare_dataframes()`; this was removed in 3.0+. Use `RegressionTest.compare()` instead.

fix

Replace `compare_dataframes(df1, df2, key)` with `RegressionTest().compare(df1, df2, key=key)`.

affects: >=3.0.0

gotchaThe key column must have unique values in each DataFrame; non-unique keys will raise an error.

fix

Ensure key column contains unique values, or use additional columns to form a composite key.

affects: all

gotchaBoth DataFrames must have identical column order; differences in column order cause match failures.

fix

Use `df.select(*ordered_columns)` before comparing to enforce column order.

affects: all

deprecatedThe `ignore_nulls` parameter is deprecated starting 4.0.0 and will be removed; use `allow_null_mismatch` instead.

fix

Replace `ignore_nulls=True` with `allow_null_mismatch=True`.

affects: >=4.0.0

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'pyspark_regression'

The package is installed as pyspark-regression but import uses underscore.

fix

Install with pip install pyspark-regression and import using from pyspark_regression import RegressionTest.

AttributeError: module 'pyspark_regression' has no attribute 'RegressionTest'

Using wrong import path; RegressionTest is not in a submodule.

fix

Use from pyspark_regression import RegressionTest directly.

pyspark.errors.exceptions.capture.SparkException: Found duplicate keys in DataFrame

Key column contains non-unique values.

fix

Drop duplicates or use a different key/combination of columns that yields unique rows.

Upgrade

Version history

4.2.4latest on PyPI · released Mar 17, 2026

Audit

Dependencies

pysparkrequiredCore dependency for Spark DataFrame operations.

Agent activity

9 hits · last 30 days

node

Amazon

Resources

githubhttps://github.com/forrest-bajbek/pyspark-regression ↗

packagehttps://pypi.org/project/pyspark-regression/ ↗

docshttps://forrest-bajbek.github.io/pyspark-regression ↗