Registry /
workflow / apache-airflow-providers-apache-spark
Install & Compatibility
Where this runs
tested against v6.3.2 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.915 runs
installs and imports cleanly · install 0.1s · import 6.283s · 619MB
glibcpy 3.10–3.915 runs
installs and imports cleanly · install 47.8s · import 5.864s · 587MB
722MB installed
● package 722MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
SparkSubmitOperator
✓ from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator
SparkSqlOperator
✓ from airflow.providers.apache.spark.operators.spark_sql import SparkSqlOperator
PySparkOperator
✓ from airflow.providers.apache.spark.operators.pyspark import PySparkOperator
SparkJDBCOperator
✓ from airflow.providers.apache.spark.operators.spark_jdbc import SparkJDBCOperator
This example demonstrates a basic Airflow DAG using the `SparkSubmitOperator` to submit a PySpark application to a Spark cluster. Before running, ensure you have a 'Spark' connection (e.g., `spark_default`) configured in your Airflow UI with the appropriate Spark master URL. The `application` parameter should point to your PySpark script accessible by the Airflow worker.
from __future__ import annotations
import pendulum
from airflow.models.dag import DAG
from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator
# For local testing, ensure a Spark Connection 'spark_default' is configured in Airflow UI.
# Example: Host: spark://localhost:7077 (or similar Spark Master URL)
# For a PySpark job, you might need a local 'pyspark_job.py' file.
# Example pyspark_job.py content:
# from pyspark.sql import SparkSession
# spark = SparkSession.builder.appName('SimpleSparkApp').getOrCreate()
# data = [('Alice', 1), ('Bob', 2), ('Charlie', 3)]
# df = spark.createDataFrame(data, ['Name', 'Age'])
# df.show()
# spark.stop()
with DAG(
dag_id="spark_submit_example_dag",
start_date=pendulum.datetime(2023, 1, 1, tz="UTC"),
catchup=False,
schedule=None,
tags=["spark", "example"],
) as dag:
submit_pyspark_job = SparkSubmitOperator(
task_id="submit_pyspark_job",
conn_id="spark_default", # Ensure this Spark connection is configured in Airflow UI
application="/opt/airflow/dags/pyspark_job.py", # Path to your PySpark script
name="airflow_pyspark_job",
conn_id="spark_default",
conf={
"spark.executor.memory": "2g",
"spark.driver.memory": "1g"
},
verbose=True,
# For more options, see SparkSubmitOperator documentation
# application_args=["--input", "/path/to/input.csv", "--output", "/path/to/output.csv"]
)
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'airflow.providers.apache.spark'
The 'apache-airflow-providers-apache-spark' package is not installed.
fixInstall the package using 'pip install apache-airflow-providers-apache-spark'.
Exception when importing 'airflow.providers.apache.spark.hooks.spark_jdbc.SparkJDBCHook' from 'apache-airflow-providers-apache-spark' package: name 'client' is not defined
The 'kubernetes' Python package is missing, which is required by the Spark provider.
fixInstall the 'kubernetes' package using 'pip install kubernetes'.
Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
The Spark cluster lacks available resources to run the job.
fixEnsure that the Spark cluster has sufficient resources and that workers are properly registered.
ModuleNotFoundError: No module named 'airflow.providers.apache'
The `apache-airflow-providers-apache-spark` package, or its parent `apache` provider directory, is not installed or not accessible within the Airflow environment where the DAG is being parsed or executed.
fixEnsure the provider package is installed correctly in the Airflow environment (e.g., `pip install apache-airflow-providers-apache-spark`). If using Docker, rebuild the Docker image after adding the installation command to the Dockerfile.
airflow.exceptions.AirflowException: Cannot execute: spark-submit ... Error code is: ...
This error often indicates that the `spark-submit` command is not found in the PATH of the Airflow worker, or there's an issue with the Spark installation or the application itself that prevents `spark-submit` from executing successfully.
fixVerify that the `spark-submit` binary is available in the system's PATH where the Airflow worker is running. Ensure Spark is correctly installed and configured, and check the full Spark logs for more specific errors.
Upgrade
Version history
6.3.2latest on PyPI · released Aug 23, 2026
Audit
Dependencies
apache-airflowrequiredCore Airflow functionality is required. Version 6.x.x of this provider requires Airflow >=2.11.0.
pyspark-clientrequiredRequired for Spark Connect functionality. Minimum version 4.0.0.
grpcio-statusrequiredRequired for Spark Connect functionality. Minimum version 1.67.0.
pysparkoptionalOptional. Required if using Spark connection types other than 'spark-connect'. No longer included by default since 6.0.0.
apache-airflow-providers-cncf-kubernetesoptionalOptional. Required for submitting Spark jobs to Kubernetes.