Install & Compatibility
Where this runs
tested against v0.29.20 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.95 runs
installs and imports cleanly · install 0.0s · import 3.114s · 151.1MB
glibcpy 3.10–3.95 runs
installs and imports cleanly · install 14.6s · import 2.884s · 147MB
152MB installed
● package 152MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
celery_executor
✓ from dagster_celery import celery_executor
Main executor for running Dagster ops on a Celery cluster.
celery_k8s_executor
✓ from dagster_celery import celery_k8s_executor
Specialized executor for running Dagster ops on a Celery cluster within Kubernetes.
This quickstart demonstrates how to configure a Dagster job to use the Celery executor. It defines a simple op and a job, then configures `celery_executor` with basic broker and backend URLs (using environment variables for flexibility). For this to run, you need a running Celery broker/backend (e.g., Redis) and a Celery worker started with `celery -A your_module worker -l info` pointing to your Dagster code. The job will then submit its op execution to the Celery queue.
import os
from dagster import Definitions, job, op
from dagster_celery import celery_executor
# Configure Celery broker and backend URLs via environment variables
# Example for local development with Redis: export CELERY_BROKER_URL="redis://localhost:6379/0"
# Example for local development with Redis: export CELERY_RESULT_BACKEND="redis://localhost:6379/0"
celery_broker_url = os.environ.get('CELERY_BROKER_URL', 'redis://localhost:6379/0')
celery_result_backend = os.environ.get('CELERY_RESULT_BACKEND', 'redis://localhost:6379/0')
celery_config = {
"broker_url": celery_broker_url,
"result_backend": celery_result_backend,
"task_default_queue": "dagster_celery_queue", # Optional: dedicated queue for Dagster tasks
}
@op
def my_celery_op(context):
context.log.info("Executing op on Celery worker!")
return "Hello from Celery!"
@job(executor_def=celery_executor.configured(celery_config))
def my_celery_job():
my_celery_op()
# To run this, you would need to:
# 1. Start a Redis server (or other message broker/backend).
# 2. Start a Celery worker: celery -A dagster_celery.app worker -l info -Q dagster_celery_queue
# (Assuming your Dagster code is in a module named 'dagster_celery.app' or similar)
# 3. Load the repository with 'dagster dev' and launch a run for 'my_celery_job'.
defs = Definitions(
jobs=[my_celery_job]
)
Debug
Known issues
gotchaThe `dagster-celery` library requires an externally provisioned and managed Celery cluster (broker, backend, and workers). Dagster itself does not manage the lifecycle of Celery components. Misconfiguration of `broker_url` or `result_backend` in `celery_config`, or failure to start Celery workers, are common sources of errors.fixEnsure your Celery broker and result backend URLs are correct and accessible from both Dagster and Celery workers. Verify Celery workers are running and configured to listen on the correct queues (e.g., `task_default_queue`). Use `celery inspect ping` to check worker connectivity.
affects: All versions
breakingDagster's execution engine configuration has evolved across major versions. Older patterns involving `config_schema` or directly passing configuration dictionaries to `executor_def` might be deprecated or behave differently. Ensure you are using the `executor_def.configured(config)` pattern.fixAlways use the `executor_def=celery_executor.configured(celery_config)` pattern when defining your jobs or `Definitions` object. Refer to the official Dagster documentation for the specific version you are using.
affects: Before 1.0.0, and some minor versions after.
gotchaVersion compatibility between `dagster` core and `dagster-celery` is critical. Mismatched major versions can lead to runtime errors or unexpected behavior due to API changes in the core execution engine.fixAlways install `dagster-celery` with a version that matches your `dagster` core version's major and minor numbers. For example, if `dagster` is `1.13.0`, `dagster-celery` should be `0.29.0` (as per the release notes, library versions are `core_minor_version - 100` for the major part and `core_patch_version` for the minor part after Dagster 1.0).
affects: All versions
gotchaWhen using `celery_k8s_executor`, the Kubernetes cluster must be properly configured for Celery workers, including appropriate images, resource requests/limits, environment variables, secrets (for broker/backend), and persistent storage if needed. Issues with any of these can prevent ops from running or reporting results.fixThoroughly review the `dagster-celery` Kubernetes documentation. Ensure your Celery worker deployment correctly references the image containing your Dagster code, has network access to the broker/backend, and possesses necessary permissions and resources within Kubernetes.
affects: All versions using `celery_k8s_executor`
Errors
Common errors & fixes
kombu.exceptions.OperationalError: [Errno 111] Connection refused
The Celery worker cannot connect to the configured message broker (e.g., RabbitMQ, Redis) because the broker is either not running, inaccessible from the worker's environment, or the `broker_url` is incorrect.
fixEnsure your Celery broker is running and network-accessible to the Dagster-Celery workers. Verify that the `broker_url` in your Celery executor configuration is correctly specified. For RabbitMQ, you might start it with `docker run -p 5672:5672 rabbitmq:3.8.2` or check your existing broker service status.
CheckError('Invariant failed. Description: Could not load run <run_id>')
The Dagster-Celery worker is unable to load the necessary run metadata or event logs from the Dagster instance, typically because the run and event log storage is not persistently configured or is not shared and synchronized across the Dagster webserver/daemon and all Celery workers.
fixConfigure a persistent and shared storage solution (e.g., PostgreSQL for `PostgresRunStorage` and `PostgresEventLogStorage`) for your Dagster instance in `dagster.yaml`, ensuring that both the Dagster webserver/daemon and all Celery workers point to and can access this same central storage.
ModuleNotFoundError: No module named 'dagster_k8s'
This error occurs when the `dagster-celery` executor is configured to use Kubernetes-specific features (e.g., `celery_k8s_job_executor`) but the `dagster-k8s` library, which provides these functionalities, is not installed or available in the Python environment where the Celery worker is running.
fixInstall the `dagster-k8s` package in the environment where your Celery workers are executing tasks: `pip install dagster-k8s`. If using Docker, ensure `dagster-k8s` is included in your worker's Docker image.
WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL)
A Celery worker process was forcibly terminated by the operating system, most commonly due to exceeding its allocated memory (Out-Of-Memory, OOM) or hitting other system resource limits, indicating a potential memory leak or insufficient resources for the executed tasks.
fixMonitor your Celery workers' memory and CPU usage to identify resource bottlenecks. Consider optimizing your Dagster ops to reduce memory consumption, increase the resources allocated to your Celery workers, or configure Celery to split large tasks into smaller, more manageable units.
Upgrade
Version history
0.29.20latest on PyPI · released Aug 27, 2026
Audit
Dependencies
dagsterrequiredCore Dagster library, required for defining jobs and ops, which dagster-celery executes.