Install & Compatibility
Where this runs
tested against v4.1.2 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.940 runs
installs and imports cleanly · install 0.0s · import 0.571s · 399.1MB
glibcpy 3.10–3.940 runs
installs and imports cleanly · install 27.0s · import 0.509s · 374MB
385MB installed
● package 385MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
SparkSession
✓ from pyspark.sql import SparkSession
✗ from pyspark.sql.connect.session import SparkSession
The `SparkSession` is imported from `pyspark.sql` and then configured for remote connection using `.remote()`.
This quickstart demonstrates how to establish a connection to a Spark Connect server and perform basic DataFrame operations. It assumes a Spark Connect server is already running and accessible at the specified URL (defaulting to `sc://localhost:15002`). The `SPARK_CONNECT_SERVER_URL` environment variable can be used to override the connection string.
import os
from pyspark.sql import SparkSession
from pyspark.sql.functions import lit
# Ensure a Spark Connect server is running, e.g., via ./sbin/start-connect-server.sh
# The default address is sc://localhost:15002
# Connect to the Spark Connect server
spark = SparkSession.builder.remote(os.environ.get('SPARK_CONNECT_SERVER_URL', 'sc://localhost:15002')).getOrCreate()
# Create a DataFrame
df = spark.range(10).withColumn("hello", lit("world"))
# Show the DataFrame
df.show()
# Perform a simple operation
result = df.filter(df.id > 5).count()
print(f"Count of rows with id > 5: {result}")
spark.stop()
Debug
Known issues
gotchaSpark Connect operates on a decoupled client-server architecture. This means your client application does not run in the same JVM process as the Spark driver. Consequently, direct access to the underlying Java Virtual Machine (JVM) objects (e.g., `df._jdf`) via Py4J, common in traditional PySpark, is not possible.fixRewrite code to use standard PySpark DataFrame API methods. Avoid direct JVM object manipulation. Focus on the DataFrame API and logical plans.
affects: Spark 3.4.0+ (all Spark Connect versions)
gotchaThe `pyspark-client` is a client library only. It does not include or automatically start a Spark cluster or Spark Connect server. You must have a Spark Connect server running and accessible (e.g., via `start-connect-server.sh` from a full Spark distribution) before your client application can connect.fixEnsure a Spark Connect server is deployed and running, and configure your client's `SparkSession.builder.remote()` method with the correct connection string (e.g., `sc://localhost:15002`).
affects: All versions of `pyspark-client`
breakingStarting with Spark 4.0, ANSI SQL mode is enabled by default (`spark.sql.ansi.enabled` set to `true`). This changes how SQL operations handle invalid or undefined results. Operations that previously returned `NULL` (e.g., division by zero, invalid casts) will now throw runtime exceptions.fixReview existing SQL queries and DataFrame operations that might rely on `NULL` handling. Implement explicit error handling or adjust queries to conform to ANSI SQL standards. The legacy behavior can be restored by setting `spark.sql.ansi.enabled` to `false`.
affects: Spark 4.0.0+ (including `pyspark-client` 4.0.0+)
breakingPySpark 4.1 (and thus `pyspark-client` 4.1.1) drops support for Python 3.9. Additionally, minimum required versions for `pyarrow` and `pandas` have been raised to `pyarrow>=15.0.0` and `pandas>=2.2.0`.fixUpgrade your Python environment to 3.10 or later. Update `pyarrow` and `pandas` to their respective minimum required versions or newer. If using `pyspark[connect]`, `pip` will generally handle these dependencies correctly.
affects: PySpark 4.1.0+ / `pyspark-client` 4.1.0+
gotchaIn Spark 4.1, `DataFrame['name']` on the Spark Connect Python Client no longer eagerly validates the column name. This means misspelled or non-existent column names might not raise an error until later in execution.fixSet the environment variable `PYSPARK_VALIDATE_COLUMN_NAME_LEGACY=1` to restore the legacy eager validation behavior if desired.
affects: Spark 4.1.0+ (for Spark Connect Python Client)
Upgrade
Version history
4.1.2latest on PyPI · released May 21, 2026
Audit
Dependencies
pysparkrequiredThe client is part of the PySpark ecosystem and typically used alongside a full PySpark installation, though `pyspark-client` can be installed standalone for just the client. `pyspark[connect]` provides the client and its dependencies.
grpciooptionalUnderpins the communication protocol between the client and the Spark Connect server.
pyarrowoptionalUsed for optimized, columnar data transfer between the Spark Connect server and the client.