Registry / data / snakemake-interface-storage-plugins

snakemake-interface-storage-plugins

JSON →
library4.4.1pypypi✓ verified 85d ago

This package provides a stable and consistent interface for developers to build custom storage plugins that integrate with Snakemake. It defines the abstract base classes and helper functions necessary for Snakemake to interact with various storage backends. The current version is 4.4.1, with a release cadence of roughly every 1-3 months, primarily focusing on new features, bug fixes, and compatibility with Snakemake core.

pip install snakemake-interface-storage-plugins
INSTALL
IMPORT
SIG · SNAKEMAKE-INTERFAC
S
snakemake-interface-storage-plugins
datapythonv4.4.1
Install
2.3s avg
Import
253ms
Disk
18MB
Pass rate
10/ 10
Env Coverage10 / 10
glibc
3.93.13
musl
3.93.13
Install & Compatibility
Where this runs
tested against v4.4.1 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
musl
py 3.103.920 runs
installs and imports cleanly · install 0.0s · import 0.271s · 20.5MB
glibc
py 3.103.920 runs
installs and imports cleanly · install 2.3s · import 0.234s · 21MB
18MB installed
● package 18MB
Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

StorageProviderBase
from snakemake_interface_storage_plugins.storage_provider import StorageProviderBase
StorageObject
from snakemake_interface_storage_plugins.storage_object import StorageObject
StorageProviderSettingsBase
from snakemake_interface_storage_plugins.settings import StorageProviderSettingsBase
StoragePlugin
from snakemake_interface_storage_plugins.registry import StoragePlugin
from snakemake_interface_storage_plugins.plugin import StoragePlugin
The StoragePlugin class for registering plugins was moved to the 'registry' submodule.

This quickstart demonstrates the core components required to implement a minimal Snakemake storage plugin. You need to define a `StorageObject` (representing a file/directory on your custom storage) and a `StorageProviderBase` (which provides `StorageObject` instances). Finally, register your plugin using `StoragePlugin` to make it discoverable by Snakemake.

from snakemake_interface_storage_plugins.storage_object import StorageObject from snakemake_interface_storage_plugins.storage_provider import StorageProviderBase from snakemake_interface_storage_plugins.registry import StoragePlugin from typing import Optional # 1. Define your custom StorageObject, implementing required methods class MyMinimalStorageObject(StorageObject): def exists(self) -> bool: # Placeholder: In a real plugin, check if the remote object exists. return True def mtime(self) -> float: return 0.0 # Last modified time def size(self) -> int: return 0 # Size in bytes def download_obj(self) -> None: pass # Download object to local path def upload_obj(self) -> None: pass # Upload local object to remote def list_local_files(self) -> list[str]: return [] # List files in local path def cleanup(self) -> None: pass # Cleanup temporary files # 2. Define your custom StorageProvider class MyMinimalStorageProvider(StorageProviderBase): def get_storage_object(self, url: str) -> StorageObject: # Return an instance of your custom StorageObject for the given URL. # 'query' and 'protocol' are typically parsed from the URL. return MyMinimalStorageObject(query=None, protocol=self.name, path=url) def example_path(self, protocol: Optional[str] = None) -> str: # Provide an example path for your protocol (e.g., for documentation). return f"{self.name}://path/to/file.txt" @property def name(self) -> str: # The unique name for your storage protocol (e.g., 's3', 'gcs', 'my-minimal'). return "my-minimal" # 3. Register your plugin with Snakemake # This makes your plugin discoverable by Snakemake when it parses Snakefiles. StoragePlugin( name="my-minimal", storage_provider=MyMinimalStorageProvider, # settings_cls=None # Optionally register a custom settings class ) print("Minimal Snakemake storage plugin 'my-minimal' defined and registered.") # To make this plugin active for Snakemake, save this code in a Python file # (e.g., my_storage_plugin.py) and ensure it's on your PYTHONPATH or installed # as part of a Snakemake extension package.
Debug
Known issues
breakingThe `checksum` attribute was added to `IOCacheInterface` and `StorageObject` in v4.4.0. Existing custom storage plugins might need to be updated to properly handle checksum generation and validation to maintain full compatibility and leverage new Snakemake features.
fix
Review `IOCacheInterface` and `StorageObject` definitions. Implement checksum handling methods/attributes in your custom `StorageObject` and `IOCacheInterface` (if used) to avoid potential future compatibility issues or `TypeError`s when Snakemake expects this functionality.
affects: >=4.4.0
gotchaOlder versions of `snakemake-interface-storage-plugins` (prior to v4.3.3) did not explicitly list `humanfriendly` as a direct dependency. If `humanfriendly` is not installed by another package, you might encounter a `ModuleNotFoundError`.
fix
Ensure `humanfriendly` is installed in your environment: `pip install humanfriendly`. Upgrade to version 4.3.3 or newer to have it automatically included as a dependency.
affects: <4.3.3
breakingChanges in error handling, specifically around `FileOrDirectoryNotFoundError` and its conversion to `WorkflowError`, were introduced in versions 4.3.0 and 4.3.1. Custom plugins raising specific exceptions might need adjustment.
fix
Review how your custom plugin handles file/directory not found scenarios and other I/O errors. Ensure that custom exceptions are handled or converted appropriately to align with the interface's expected error types, especially `WorkflowError` for user-facing issues.
affects: >=4.3.0
gotchaPython 3.10 compatibility fixes were applied in v4.4.1. If you are using Python 3.10 with versions prior to 4.4.1, you might encounter unexpected runtime errors or incorrect behavior.
fix
Upgrade `snakemake-interface-storage-plugins` to v4.4.1 or higher when using Python 3.10 to ensure full compatibility: `pip install --upgrade snakemake-interface-storage-plugins`.
affects: <4.4.1
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'humanfriendly'
The `humanfriendly` package, used internally by the storage plugin interface, was not installed as a direct dependency in versions prior to v4.3.3.
fix
Install the missing dependency: `pip install humanfriendly`. Alternatively, upgrade to `snakemake-interface-storage-plugins>=4.3.3` which includes `humanfriendly` as a direct dependency.
TypeError: Can't instantiate abstract class MyCustomStorageObject with abstract methods exists, mtime, size, download_obj, upload_obj, list_local_files, cleanup
Your custom `StorageObject` class inherits from `snakemake_interface_storage_plugins.storage_object.StorageObject` but has not implemented all of its required abstract methods.
fix
Ensure that your `MyCustomStorageObject` class (and any custom `IOCacheInterface` if used) implements all abstract methods defined in the `StorageObject` base class. You must provide concrete implementations for `exists`, `mtime`, `size`, `download_obj`, `upload_obj`, `list_local_files`, and `cleanup`.
TypeError: __init__() missing 1 required positional argument: 'checksum'
You are trying to instantiate `StorageObject` or `IOCacheInterface` (or a custom class inheriting from them) without providing the `checksum` argument, which became required from v4.4.0 onwards for full functionality.
fix
Update your code to provide a `checksum` argument when initializing `StorageObject` or `IOCacheInterface` instances, or ensure your custom plugin adheres to the updated interface. For example: `MyStorageObject(query=None, protocol='my', path=url, checksum=None)` (if checksum is optional for your specific case, otherwise provide a valid checksum value).
Upgrade
Version history
4.4.1latest on PyPI · released Mar 16, 2026
Audit
Dependencies
humanfriendlyrequiredUsed for human-readable file sizes and other utilities within the interface. It became a direct dependency from v4.3.3 onwards.
Agent activity
25 hits · last 30 days
node
20
OpenAI (training)
1
Resources
snakemake-interface-storage-plugins — pip install snakemake-interface-storage-plugins · libregistry