Registry / aws / multi-storage-client

multi-storage-client

JSON →
library0.50.0pypypi✓ verified 85d ago

The Multi-Storage Client (MSC) is a unified, high-performance Python client designed for seamless interaction with various object and file stores, including AWS S3, Azure Blob Storage, Google Cloud Storage (GCS), NVIDIA AIStore, Oracle Cloud Infrastructure (OCI) Object Storage, and POSIX file systems. It offers a generic interface, simplifying data access and enabling storage location changes without extensive code modifications. As of its current version 0.46.0, it maintains an active release cadence with frequent updates.

pip install multi-storage-client
INSTALL
IMPORT
SIG · MULTI-STORAGE-CLIE
M
multi-storage-client
awspythonv0.50.0
Install
6.4s avg
Import
1124ms
Disk
75MB
Pass rate
10/ 10
Env Coverage10 / 10
glibc
3.93.13
musl
3.93.13
Install & Compatibility
Where this runs
tested against v0.23.0 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
musl
py 3.103.980 runs
installs and imports cleanly · install 0.0s · import 1.056s · 62.3MB
glibc
py 3.103.980 runs
installs and imports cleanly · install 6.4s · import 1.191s · 90MB
75MB installed
● package 75MB
Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

StorageClient
from multistorageclient.client import StorageClient
msc_path
from multistorageclient.path import msc_path

This quickstart demonstrates how to initialize the `StorageClient` using a YAML configuration file and perform basic file operations (write, read, list) with a local POSIX storage profile. Authentication for cloud providers is typically managed within the YAML profiles or via environment variables, avoiding hardcoding in Python code.

import os import yaml from multistorageclient.client import StorageClient # Create a dummy MSC configuration file for local testing config_content = ''' profiles: local-files: storage_provider: type: posix options: base_path: ./msc_data ''' # Create a directory for local files os.makedirs('./msc_data', exist_ok=True) config_path = './msc_config.yaml' with open(config_path, 'w') as f: f.write(config_content) # Set the MSC_CONFIG environment variable to point to the config file os.environ['MSC_CONFIG'] = config_path try: # Instantiate the client client = StorageClient() # Example: Write a file using the 'local-files' profile local_file_path = 'msc://local-files/example.txt' with client.open(local_file_path, 'w') as f: f.write('Hello from Multi-Storage Client!') print(f"Content written to {local_file_path}") # Example: Read the file with client.open(local_file_path, 'r') as f: content = f.read() print(f"Content read from {local_file_path}: {content}") # Example: List objects in the 'local-files' profile objects = client.list_objects('msc://local-files/') print("Objects in 'msc://local-files/':") for obj in objects: print(f"- {obj.uri}") finally: # Clean up the dummy config file and data directory if os.path.exists(config_path): os.remove(config_path) if os.path.exists('./msc_data/example.txt'): os.remove('./msc_data/example.txt') if os.path.exists('./msc_data'): os.rmdir('./msc_data') del os.environ['MSC_CONFIG']
Debug
Known issues
breakingPython 3.9 support was dropped in version 0.43.0. Users on Python 3.9 must upgrade to Python 3.10 or newer.
fix
Upgrade your Python environment to version 3.10 or later. The library requires Python <4, >=3.10.
affects: >=0.43.0
gotchaThe Multi-Storage Client heavily relies on external YAML configuration files for defining storage profiles and credentials. Direct in-code configuration for providers/credentials is not the primary pattern.
fix
Define your storage provider profiles (including credentials and options) in a YAML file and set the `MSC_CONFIG` environment variable to its path.
affects: >=0.1.0
gotchaTo support specific cloud storage providers (e.g., AWS S3, Google Cloud Storage), you must install the corresponding optional dependencies using `pip install "multi-storage-client[extra_name]"`. The base `multi-storage-client` package only supports POSIX file systems by default.
fix
Install the required extras for your cloud provider, e.g., `pip install "multi-storage-client[boto3]"` for AWS S3.
affects: >=0.1.0
gotchaAttempting to import `multistorageclient.instrumentation.auth` without installing the `[vault]` extra will result in an `ImportError`.
fix
If you intend to use `vault` integration, install it via `pip install "multi-storage-client[vault]"`. Otherwise, avoid importing the instrumentation.auth module if the extra is not installed.
affects: >=0.45.0
deprecatedThe `cryptography` package was incorrectly made a core requirement in version 0.45.0. This was corrected in a subsequent release (0.45.1) to restore it as an optional dependency, reducing the default install footprint for users not needing its features.
fix
Upgrade to version 0.45.1 or newer to ensure `cryptography` is an optional dependency. If you require `cryptography` features, ensure you install the relevant extra (e.g., `[vault]`).
affects: 0.45.0
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'multi_storage_client.providers.s3'
The base `multi-storage-client` package only supports POSIX file systems by default; specific cloud storage providers require installing their corresponding optional dependencies.
fix
Install the required extra for your cloud provider, e.g., `pip install "multi-storage-client[boto3]"` for AWS S3, `"multi-storage-client[azure-storage-blob]"` for Azure Blob Storage, or `"multi-storage-client[google-cloud-storage]"` for Google Cloud Storage.
ImportError: cannot import name 'auth' from 'multi_storage_client.instrumentation'
Attempting to import the `auth` module from `multi_storage_client.instrumentation` without installing the `[vault]` extra will result in an `ImportError`.
fix
If you intend to use `vault` integration, install it via `pip install "multi-storage-client[vault]"`. Otherwise, avoid importing this specific module if the extra is not installed.
Python 3.9 support was dropped in version 0.43.0
The `multi-storage-client` library versions 0.43.0 and newer require Python 3.10 or later, so users on Python 3.9 will encounter incompatibility.
fix
Upgrade your Python environment to version 3.10 or later.
ValueError: This library only supports credentials from google-auth-library-python.
When using `google.cloud.storage.Client`, an older or incorrect method for providing credentials (e.g., `GoogleCredentials.from_stream`) is being used instead of the `google-auth-library-python` supported methods.
fix
Authenticate using `google-auth-library-python` methods, such as `storage.Client.from_service_account_json('/path/to/keyfile.json')` or by relying on Application Default Credentials (ADC) by setting `GOOGLE_APPLICATION_CREDENTIALS` environment variable or using `storage.Client()`.
Connection refused (Connection refused)
This error, often seen when connecting to S3 or other cloud storage, typically indicates a network connectivity issue, incorrect endpoint configuration, or firewall restrictions preventing the client from reaching the storage service.
fix
Verify that your storage provider configuration (e.g., `region_name`, endpoint URL) is correct in your MSC configuration file, check network connectivity from your client to the storage service, and ensure no firewalls are blocking the connection.
Upgrade
Version history
0.50.0latest on PyPI · released Jun 5, 2026
Audit
Dependencies
boto3optionalRequired for AWS S3 and S3-compatible object storage support.
google-cloud-storageoptionalRequired for Google Cloud Storage (GCS) support.
azure-storage-bloboptionalRequired for Azure Blob Storage support.
ocioptionalRequired for Oracle Cloud Infrastructure (OCI) Object Storage support.
aistoreoptionalRequired for NVIDIA AIStore support.
cryptographyoptionalUsed for `vault` extra, incorrectly added as core dependency in 0.45.0, restored as optional.
Agent activity
21 hits · last 30 days
node
20
OpenAI (training)
1
Resources
multi-storage-client — pip install multi-storage-client · libregistry