Install & Compatibility
Where this runs
tested against v0.29.9 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.920 runs
installs and imports cleanly · install 0.0s · import 2.887s · 131.5MB
glibcpy 3.10–3.920 runs
installs and imports cleanly · install 12.6s · import 2.755s · 127MB
134MB installed
● package 134MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
create_spark_op
✓ from dagster_spark import create_spark_op
define_spark_config
✓ from dagster_spark import define_spark_config
SparkDeclarativePipelineComponent
✓ from dagster_spark.components.spark_declarative_pipeline import SparkDeclarativePipelineComponent
This API is currently in feature preview and not considered ready for production use; it may have breaking changes in patch releases.
This quickstart defines a Dagster job that executes a simple Spark application (like SparkPi) using `create_spark_op`. It demonstrates how to define Spark configuration and wrap a Spark job for orchestration within Dagster. Replace `path/to/your/spark-examples.jar` with the actual path to your Spark application JAR.
from dagster import job, Definitions, asset
from dagster_spark import create_spark_op, define_spark_config
# Define Spark configuration
my_spark_config = define_spark_config(
{
"spark.master": "local[*]",
"spark.app.name": "dagster-spark-example"
}
)
# Create an op from the Spark job definition
my_spark_job_op = create_spark_op(
main_class="org.apache.spark.examples.SparkPi",
jars=["path/to/your/spark-examples.jar"], # Replace with your Spark job JAR path
spark_config=my_spark_config,
name="my_spark_pi_op"
)
@job
def spark_pi_job():
my_spark_job_op()
# Or, integrate with assets (requires PySparkResource from dagster-pyspark for direct SparkSession)
# For a simple asset that just runs a Spark job via `spark-submit`, you might do:
# @asset
# def spark_data_asset():
# # This would involve using create_spark_op within an asset or using Dagster Pipes
# # For simplicity, this example focuses on a job.
# pass
defs = Definitions(jobs=[spark_pi_job])
dagster --version
Debug
Known issues
breakingThe `SparkSolidDefinition` has been removed. Users should migrate to `create_spark_op` for defining Spark-based operations.fixReplace `SparkSolidDefinition` with `create_spark_op`.
affects: <=0.6.0
deprecatedSpark Step Launchers are superseded by Dagster Pipes and are no longer the recommended method for launching external code from Dagster ops and assets. While still available, they will not receive new features or active development.fixConsider migrating to Dagster Pipes for more lightweight and flexible Spark job orchestration, especially for new projects or feature development.
affects: All versions
gotchaThe Spark Declarative Pipeline (SDP) integration components (`SparkDeclarativePipelineComponent`, `SparkPipelinesResource`) are in feature preview. This API may have breaking changes in patch version releases and is not considered ready for production use.fixUse with caution in production environments. Monitor Dagster release notes for updates on SDP API stability.
affects: 0.28.21 and later
gotchaEnsure compatibility between your `dagster-spark` version, the `pyspark` library version (if used), and your Apache Spark cluster version. Specific Hadoop/AWS Java SDK versions might also be critical for integrations like S3.fixRefer to Dagster's official documentation for recommended compatibility matrices. Test deployments thoroughly across environments.
affects: All versions
breakingDagster core (a dependency of `dagster-spark`) no longer supports Python 3.8 and requires `pydantic>=2`.fixUpgrade your Python environment to 3.10 or newer and ensure `pydantic` is version 2 or greater.
affects: Dagster 1.12.0 (Dagster Spark 0.28.21) and later
Upgrade
Version history
0.29.9latest on PyPI · released Jun 11, 2026
Audit
Dependencies
dagsterrequiredCore orchestration framework dependency.
pysparkoptionalRequired for defining and executing PySpark jobs.
pydantic>=2requiredRequired by Dagster core.