Install & Compatibility
Where this runs
tested against v0.40.2 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.95 runs
installs and imports cleanly · install 0.0s · import 0.468s · 41.6MB
glibcpy 3.10–3.95 runs
installs and imports cleanly · install 3.7s · import 0.452s · 42MB
38MB installed
● package 38MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
FSApp
✓ from dvc_task.app import FSApp
Main application class for a filesystem-based Celery app.
ProcessManager
✓ from dvc_task.proc import ProcessManager
Manages background processes within tasks.
This quickstart demonstrates how to initialize a filesystem-based Celery application (FSApp), define a task to run a background process using ProcessManager, and then start a temporary worker to process the task. It showcases how to queue a simple 'echo' command.
from dvc_task.app import FSApp
from dvc_task.proc import ProcessManager
import time
import os
# Create a filesystem-based Celery app
# For a real application, consider a more persistent wdir
app = FSApp(wdir="./dvc-task-data")
@app.task
def run_command_task(command_args, task_name):
manager = ProcessManager(wdir=os.path.join(app.wdir, 'processes'))
# Run a simple command as a background process
# The '.delay()' call queues the task
signature = manager.run(command_args, name=task_name).delay()
return signature.id
# Example usage:
if __name__ == "__main__":
print("Starting dvc-task example...")
task_id = run_command_task.delay(command_args=["echo", "Hello, dvc-task!"], task_name="greet_task")
print(f"Task 'greet_task' queued with ID: {task_id}")
print("Starting temporary worker...")
# In a real scenario, the worker would run in a separate process/daemon
# For quickstart, we run a temporary worker that exits when the queue is empty
worker = app.TemporaryWorker()
worker.start() # This blocks until the queue is empty or timeout
# Clean up (optional, but good for quickstart)
app.clean()
print("dvc-task example finished and cleaned up.")
dvc-task --version
Debug
Known issues
breakingVersion 0.3.0 introduced a breaking change by bumping the minimum required versions for Celery and Kombu to 5.3.0. Users on older versions of these dependencies will need to upgrade.fixUpgrade Celery and Kombu to at least version 5.3.0: `pip install 'celery>=5.3.0' 'kombu>=5.3.0'`.
affects: <0.3.0
gotchaUsers observed 'Celery Monitor Shutdowns' prior to version 0.40.1, indicating potential instability or unexpected behavior in monitoring. Version 0.40.1 includes a fix (dvc-10427) for this issue.fixUpgrade to dvc-task version 0.40.1 or later to resolve Celery Monitor Shutdown issues.
affects: <0.40.1
gotchaWhile `dvc-task` is tested on Windows, its upstream dependency Celery is not officially supported on Windows. This means that certain issues specific to Windows might stem from Celery itself and could have limited official support.fixBe aware of potential platform-specific issues on Windows. Refer to Celery's documentation for known Windows limitations and workarounds.
affects: All
gotchaThe version numbering jumped from `0.4.0` to `0.40.1`. While not explicitly stated as breaking, such a large jump in the minor version often signals significant changes or potential incompatibilities, even if release notes are brief. Users should review the changelog carefully when upgrading across this boundary.fixCarefully review the changelog between `0.4.0` and `0.40.1` for unlisted breaking changes or behavioral differences, and thoroughly test your application after upgrading.
affects: All versions before 0.40.1 when upgrading to 0.40.1+
Errors
Common errors & fixes
ImportError: cannot import name 'fsspec_loop' from 'fsspec.asyn'
This error occurs due to an incompatibility between the `dvc` and `dvc-task` libraries and their dependency, `fsspec`.
fixDowngrade `fsspec` to a version compatible with your `dvc` and `dvc-task` installation, for example, `pip install 'fsspec==2022.11.0'` or upgrade `dvc` and `dvc-task` to their latest versions to ensure `fsspec` compatibility.
Queued jobs failing before they get started (often with `AttributeError` in logs)
This indicates an `AttributeError` occurring within a `dvc-task` worker, preventing the execution of a queued job, possibly due to inconsistent environments or internal state corruption. The full traceback in the logs (e.g., in `.dvc/tmp/exps/celery/result` files) would provide the specific attribute issue.
fixEnsure all necessary dependencies and environment variables are consistent between where the task was queued and where the worker is executing it. Debug the failing task by trying to run it outside the queue (`dvc exp run` without `--queue`). Running `dvc exp clean` can also help clear any corrupted temporary files.
ERROR: unexpected error - cannot import name 'umask' from 'dvc_objects.fs.system'
This `ImportError` signifies a version mismatch between your `dvc` installation (which includes `dvc-task` as a subproject) and the `dvc_objects` library.
fixDowngrade `dvc-objects` to a compatible version (e.g., `pip install 'dvc-objects<=1.3.2'`) or upgrade your entire `dvc` package to its latest version to resolve the dependency conflict.
ERROR: unexpected error - Extra data: line 1 column 56 (char 55). (from `dvc queue status`)
This error points to a parsing issue, likely due to corrupted internal JSON or structured data files used by `dvc-task` to manage the experiment queue status. This can occur from abrupt process termination or concurrency issues.
fixClear the corrupted status by manually deleting the temporary experiment directory (`.dvc/tmp/exps`) or running the command `dvc exp clean`.
Upgrade
Version history
0.40.2latest on PyPI · released Oct 8, 2024
Audit
Dependencies
celeryrequiredCore task queue framework, dvc-task is built on top of it. Requires version 5.3 or later.
komburequiredMessaging library used by Celery, requires version 5.3 or later.