Install & Compatibility
Where this runs
tested against v0.8.0 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.920 runs
installs and imports cleanly · install 0.0s · import 0.000s · 30.8MB
glibcpy 3.10–3.920 runs
installs and imports cleanly · install 2.7s · import 0.000s · 31MB
29MB installed
● package 29MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
spark_session
✓ def test_example(spark_session):
pytest-spark provides fixtures that pytest auto-discovers; no direct import from 'pytest_spark' itself is typically needed in test files.
spark_context
✓ def test_example(spark_context):
pytest-spark provides fixtures that pytest auto-discovers; no direct import from 'pytest_spark' itself is typically needed in test files.
To get started, define a `spark_session` fixture in a `conftest.py` file. This fixture will automatically provide a SparkSession to your tests. Then, write tests that accept `spark_session` as an argument. You can run tests using `pytest` from your terminal.
import pytest
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, IntegerType
# conftest.py (placed in your project's root or tests directory)
@pytest.fixture(scope="session")
def spark_session():
"""
Fixture for creating a SparkSession for testing.
This SparkSession is reused across all tests in the session.
"""
spark = SparkSession.builder \
.master("local[*]") \
.appName("pytest-spark-session") \
.config("spark.driver.memory", "2g") \
.getOrCreate()
yield spark
spark.stop()
# test_example.py (a sample test file)
def test_data_frame_creation(spark_session):
schema = StructType([
StructField("name", StringType(), True),
StructField("age", IntegerType(), True)
])
data = [("Alice", 1), ("Bob", 2)]
df = spark_session.createDataFrame(data, schema)
assert df.count() == 2
assert df.columns == ["name", "age"]
assert df.collect()[0].name == "Alice"
Debug
Known issues
gotchaConfiguring `SPARK_HOME` has multiple methods (environment variable, `pytest.ini`, `--spark_home` CLI option) which are read in a specific order (CLI > `pytest.ini` > ENV). If `pyspark` is installed via pip, setting `SPARK_HOME` might not be necessary, leading to confusion or unexpected behavior if mismatched.fixPrefer installing `pyspark` via `pip` and omit `SPARK_HOME` if possible. If explicit `SPARK_HOME` is needed, use `pytest.ini` for project-level consistency or `--spark_home` for specific runs, understanding their precedence.
affects: All versions
breakingThe `spark_context` fixture is not supported when using Spark Connect functionality. If you're working with Spark 3.4+ and Spark Connect, you must use the `spark_session` fixture instead.fixUse the `spark_session` fixture for tests involving Spark Connect.
affects: 0.6.0 onwards (with Spark 3.4+)
gotchaBy default, the `spark_session` fixture creates a SparkSession with Hive support enabled. If Hive jars are not desired or cause conflicts, you can explicitly disable Hive support by adding `spark_options = spark.sql.catalogImplementation: in-memory` to your `pytest.ini`.fixAdd `spark_options = spark.sql.catalogImplementation: in-memory` under the `[pytest]` section in your `pytest.ini` to explicitly disable Hive support.
affects: All versions
gotchaComparing Spark DataFrames for equality can be challenging directly due to potential differences in row order, column order, or schema. Direct `==` comparison often fails even for logically identical DataFrames.fixFor robust DataFrame comparison, convert both expected and actual DataFrames to Pandas DataFrames, sort them by common keys (if order doesn't matter), and then use `pandas.testing.assert_frame_equal(df1.toPandas().sort_values(...), df2.toPandas().sort_values(...), check_like=True)` to ignore column order. Libraries like `chispa` also provide Spark DataFrame equality assertions.
affects: All versions
gotchaWhen running tests in parallel with `pytest-xdist`, session-scoped Spark fixtures (like `spark_session`) can interfere with each other if not properly isolated. Each parallel process might try to use the same temporary directories or resources, leading to data races or failures.fixUtilize `pytest`'s `tmp_path_factory` fixture to generate a unique temporary directory for each `pytest-xdist` worker process, and configure Spark to use these isolated directories for its local storage (e.g., `spark.local.dir`).
affects: All versions when using `pytest-xdist`
Upgrade
Version history
0.8.0latest on PyPI · released May 21, 2025
Audit
Dependencies
pytestrequiredTesting framework plugin integrates with.
pysparkrequiredCore Apache Spark Python API it facilitates testing for.
pyspark[connect]optionalRequired for Spark Connect functionality (Spark 3.4+), if used.
pyspark-connectoptionalAlternative package for Spark Connect (PySpark 4.x), if used.