The `dagster-k8s` library provides a robust integration for running Dagster with Kubernetes. It enables launching Dagster runs as Kubernetes Jobs, executing external code in Kubernetes pods directly from assets and ops using `PipesK8sClient`, and forms the foundation for Dagster's official Helm chart deployments. The library is actively maintained, with version `0.29.0` typically aligning with major releases of the core Dagster library.
pip install dagster-k8sVerified import paths — ran on the pinned version, not inferred.
This example demonstrates how to use `PipesK8sClient` to launch a Kubernetes pod from within a Dagster asset. The `PipesK8sClient` allows Dagster to receive real-time events from the initiated jobs running in Kubernetes. You would typically deploy this with a Dagster instance configured to run on Kubernetes.
Migrate usage from `k8s_job_op` to `PipesK8sClient` for launching Kubernetes pods from assets and ops.
For `k8s_job_executor`, use the `step_k8s_config` field directly on the executor definition to apply configuration to every step pod, or apply `dagster-k8s/config` tags to individual ops if fine-grained control is needed.
Ensure your deployment workflow includes a Dockerfile that copies your project, installs `dagster`, `dagster-k8s`, and any other project dependencies, and then build and push this image to an accessible container registry.
Refer to the official Dagster documentation and release notes before upgrading minor versions to identify potential breaking changes in beta APIs and adjust your code accordingly.
Check Kubernetes pod logs for the user code deployment (`kubectl logs <user-code-pod-name>`), verify the service is running and accessible (DNS resolution, port accessibility), ensure adequate CPU/memory resources for the user code pod, and review network policies.
Examine the logs of the crashing pod using `kubectl logs <pod-name>` (and `kubectl logs --previous <pod-name>` for past attempts) to identify the root cause, which could be a `ModuleNotFoundError`, `DagsterInvalidConfigError`, or application-level exception. Also, check pod events with `kubectl describe pod <pod-name>` for issues like `OOMKilled` or image pull errors.
Ensure your Dockerfile correctly copies your Dagster project and installs all necessary Python dependencies (including the project itself in editable mode, e.g., `pip install -e .`). Verify that your `dagster.yaml` or workspace definition correctly points to the module or package.
Verify that your Kubernetes deployment (e.g., Helm chart `values.yaml`) correctly configures environment variables or mounts secrets/config maps to the Dagster user code pods, and that the variable names match exactly. For Helm, ensure `envConfigMaps` or `envSecrets` are configured correctly.
This often requires a fix within the `dagster-k8s` library itself (as noted in GitHub issues) to robustly handle log streaming. For user-managed code, ensure containers are outputting valid UTF-8. If using custom log readers, implement more resilient decoding. Upgrading Dagster to a version where this issue is resolved is the primary solution.