The Multi-Storage Client (MSC) is a unified, high-performance Python client designed for seamless interaction with various object and file stores, including AWS S3, Azure Blob Storage, Google Cloud Storage (GCS), NVIDIA AIStore, Oracle Cloud Infrastructure (OCI) Object Storage, and POSIX file systems. It offers a generic interface, simplifying data access and enabling storage location changes without extensive code modifications. As of its current version 0.46.0, it maintains an active release cadence with frequent updates.
pip install multi-storage-clientVerified import paths — ran on the pinned version, not inferred.
This quickstart demonstrates how to initialize the `StorageClient` using a YAML configuration file and perform basic file operations (write, read, list) with a local POSIX storage profile. Authentication for cloud providers is typically managed within the YAML profiles or via environment variables, avoiding hardcoding in Python code.
Upgrade your Python environment to version 3.10 or later. The library requires Python <4, >=3.10.
Define your storage provider profiles (including credentials and options) in a YAML file and set the `MSC_CONFIG` environment variable to its path.
Install the required extras for your cloud provider, e.g., `pip install "multi-storage-client[boto3]"` for AWS S3.
If you intend to use `vault` integration, install it via `pip install "multi-storage-client[vault]"`. Otherwise, avoid importing the instrumentation.auth module if the extra is not installed.
Upgrade to version 0.45.1 or newer to ensure `cryptography` is an optional dependency. If you require `cryptography` features, ensure you install the relevant extra (e.g., `[vault]`).
Install the required extra for your cloud provider, e.g., `pip install "multi-storage-client[boto3]"` for AWS S3, `"multi-storage-client[azure-storage-blob]"` for Azure Blob Storage, or `"multi-storage-client[google-cloud-storage]"` for Google Cloud Storage.
If you intend to use `vault` integration, install it via `pip install "multi-storage-client[vault]"`. Otherwise, avoid importing this specific module if the extra is not installed.
Upgrade your Python environment to version 3.10 or later.
Authenticate using `google-auth-library-python` methods, such as `storage.Client.from_service_account_json('/path/to/keyfile.json')` or by relying on Application Default Credentials (ADC) by setting `GOOGLE_APPLICATION_CREDENTIALS` environment variable or using `storage.Client()`.Verify that your storage provider configuration (e.g., `region_name`, endpoint URL) is correct in your MSC configuration file, check network connectivity from your client to the storage service, and ensure no firewalls are blocking the connection.