PySpark is the Python API for Apache Spark, a unified analytics engine for large-scale data processing. It allows users to leverage Spark's powerful distributed computing capabilities, including Spark SQL, DataFrames, Structured Streaming, and MLlib, using familiar Python syntax. The library is actively maintained, with the current version being 4.1.1, and follows the release cadence of the broader Apache Spark project.
pip install pysparkVerified import paths — ran on the pinned version, not inferred.
This quickstart demonstrates how to initialize a SparkSession, create a DataFrame from Python data, display its schema and content, perform a filtering transformation, and then a grouping and aggregation. It also highlights the importance of setting `JAVA_HOME`.
Upgrade your Python environment to 3.10 or newer. PySpark 4.0+ supports Python 3.10, 3.11, and 3.12.
Upgrade Pandas in your environment: `pip install 'pandas>=1.0.5'`
Review your code for potential changes in behavior related to ANSI SQL mode if relying on Pandas API on Spark. Refer to Spark documentation for specifics on ANSI mode implications.
Understand the lazy evaluation model. Use `df.explain()` to see the execution plan without triggering computation, and be aware that actions force execution.
Avoid `.collect()` for large datasets. Use `df.show()`, `df.take(N)`, `df.limit(N).toPandas()`, or write to distributed storage for inspecting data or small samples.
Install a compatible JDK (e.g., OpenJDK 8 or 11) and set the `JAVA_HOME` environment variable to point to your JDK installation directory. For minimal environments (e.g., Alpine), ensure that required shell utilities like `bash` are also installed (`apk add bash` for Alpine), as Spark's startup scripts may depend on them.
Analyze your data's distribution and use `df.repartition(N, *columns)` or `df.coalesce(N)` before wide transformations to optimize partitioning, especially for high-cardinality columns.
Install PySpark using pip: `pip install pyspark` or ensure your environment variables (like `PYTHONPATH`) correctly point to your PySpark installation if it's not a standard `pip` install.
Set the `SPARK_HOME` environment variable to your Spark installation path. For example, on Linux/macOS: `export SPARK_HOME=/path/to/spark` or in a Python script using `import os; os.environ['SPARK_HOME'] = '/path/to/spark'`. If using `findspark`, call `findspark.init('/path/to/spark')`.Convert the DataFrame to an RDD using `.rdd` before applying RDD transformations (e.g., `df.rdd.map(...)`) or use DataFrame-specific methods like `select()`, `withColumn()`, `filter()`, `udf()` for column-wise operations, which are generally more efficient.
Include the required JAR file(s) in Spark's classpath. When running with `spark-submit`, use the `--jars` option (e.g., `spark-submit --jars postgresql-42.7.0.jar your_script.py`). When creating a `SparkSession`, configure it with `spark.jars` or `spark.driver.extraClassPath` properties.
Access the `SparkContext` object from your `SparkSession` instance to use `parallelize()`. For example, if your `SparkSession` object is named `spark`, use `spark.sparkContext.parallelize(...)`.