Install & Compatibility
Where this runs
tested against v0.25.13 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.920 runs
installs and imports cleanly · install 0.0s · import 2.605s · 124.3MB
glibcpy 3.10–3.920 runs
installs and imports cleanly · install 12.6s · import 2.469s · 120MB
130MB installed
● package 130MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
PipesSubprocessClient
✓ from dagster import PipesSubprocessClient
✗ from dagster_shell import create_shell_command_op
The `create_shell_command_op` and `create_shell_script_op` from `dagster-shell` are deprecated. Use `PipesSubprocessClient` from the `dagster` package instead.
This quickstart demonstrates how to use `PipesSubprocessClient` from the main `dagster` package to execute a shell command as a Dagster asset. This is the recommended approach as the direct shell ops in `dagster-shell` are deprecated. The `PipesSubprocessClient` handles streaming logs and events from the shell process back to Dagster.
import dagster as dg
import os
@dg.asset
def run_shell_command_asset(context: dg.AssetExecutionContext, pipes_subprocess_client: dg.PipesSubprocessClient):
# Example: Run a simple echo command
# The command is executed by PipesSubprocessClient, which pipes logs back to Dagster.
command = ["bash", "-c", "echo 'Hello from Dagster Pipes!' && sleep 1 && echo 'Done!'"]
# For a shell script file, you'd specify its path:
# shell_script_path = "./my_script.sh"
# with open(shell_script_path, "w") as f:
# f.write("#!/bin/bash\nset -eux\necho 'Executing my_script.sh'\n")
# command = ["bash", shell_script_path]
# Run the command and get results. PipesSubprocessClient streams logs and events.
result = pipes_subprocess_client.run(command=command, context=context).get_results()
# The result object contains information about the subprocess execution, e.g., return code
context.log.info(f"Shell command completed with exit code: {result.return_code}")
definitions = dg.Definitions(
assets=[run_shell_command_asset],
resources={
"pipes_subprocess_client": dg.PipesSubprocessClient(),
# pipes_subprocess_client can be configured, e.g., to pass environment variables
# "pipes_subprocess_client": dg.PipesSubprocessClient(env={'MY_VAR': 'my_value'})
},
)
dagster --version
Debug
Known issues
breakingThe `create_shell_command_op` and `create_shell_script_op` functions from `dagster-shell` have been deprecated in Dagster 1.10+ and should no longer be used.fixMigrate to using `PipesSubprocessClient` from the `dagster` package for executing shell commands and scripts. See the quickstart example for correct usage.
affects: Dagster 1.10+
gotchaWhen specifying environment variables for shell commands, older versions of `dagster-shell` (e.g., pre-1.0 and some 0.15.x versions) might *override* the entire environment instead of merging with the parent process's environment.fixEnsure you are using a recent version of Dagster and `PipesSubprocessClient` which correctly merges environments. If using older `dagster-shell` ops, explicitly merge `os.environ` with your desired `env` dictionary before passing it to the op config. Example: `{**os.environ, **my_custom_env}`. affects: <1.0 (Dagster core) and some 0.15.x versions
gotcha`dagster-shell` (and underlying Python `subprocess` calls) may encounter issues on Windows due to `preexec_fn` not being supported. This can manifest as `RuntimeError` or unexpected behavior.fixConsider running Dagster environments on Linux-based systems. If Windows is required, ensure your shell commands are simple and do not rely on `preexec_fn` functionality, or explore alternative execution mechanisms.
affects: All versions, specifically on Windows operating systems
gotchaExecuting binary files or commands with complex arguments via `dagster-shell` might lead to 'not-found' errors or unexpected behavior if UTF-8 encoding is implicitly applied during command interpretation.fixEnsure commands are correctly quoted and consider explicitly using `shlex.split` for complex commands. If issues persist with binary executables, confirm the executable path is correct and accessible, and that `dagster-shell` is not misinterpreting arguments due to encoding. Consider wrapping the binary call in a simple bash script.
affects: Specific versions, e.g., Dagster 1.3.13 / dagster-shell 0.19.13
Errors
Common errors & fixes
ImportError: cannot import name 'create_shell_command_op' from 'dagster_shell'
Attempting to import deprecated functions `create_shell_command_op` or `create_shell_script_op` from the `dagster-shell` library after Dagster version 1.10. These functions have been removed.
fixUpdate your code to use `PipesSubprocessClient` from the `dagster` package instead. The import should be `from dagster import PipesSubprocessClient`, and usage will involve configuring it as a resource and calling its `run` method.
RuntimeError: preexec_fn is not supported on Windows platforms
The underlying `subprocess` call used by `dagster-shell` attempts to use `preexec_fn`, a Unix-specific feature, on a Windows operating system.
fixRun your Dagster instance and code on a Linux-based environment or container. If a Windows environment is unavoidable, you may need to find alternative ways to execute shell commands that do not rely on `preexec_fn` or implement custom execution logic.
Shell command execution failed with output: [output_logs] and return code: 127 (or other non-zero code), typically for binary executables or complex commands.
The shell command or binary executable could not be found, or its arguments were misinterpreted, possibly due to encoding issues or incorrect paths. Return code 127 usually indicates 'command not found'.
fixVerify that the executable or script is in the system's PATH or provide its absolute path. Ensure complex arguments are properly quoted. If the command involves a binary, consider wrapping it in a simple bash script to explicitly control execution and argument passing.
Upgrade
Version history
0.25.13latest on PyPI · released Feb 11, 2025
Audit
Dependencies
dagsterrequiredThis library is an integration for the Dagster orchestration framework and requires `dagster` itself.
pythonrequiredRequires Python versions between 3.9 and 3.12 (exclusive of 3.13).