Registry / aws / cloudpathlib

cloudpathlib

library0.25.0pypypi✓ verified 26d ago

cloudpathlib provides pathlib-style classes for interacting with files and directories in various cloud storage services such as AWS S3, Google Cloud Storage, and Azure Blob Storage. It aims to offer a familiar filesystem interface, abstracting away cloud-specific details. The library is actively maintained, with a typical release cadence that includes bug fixes and new features. The current version is 0.23.0.

aws gcp azure

pip install cloudpathlib

github ↗package ↗homepage ↗docs ↗changelog ↗

INSTALL

IMPORT

SIG · CLOUDPATHLIB

cloudpathlib

awspythonv0.25.0

Install

4.5s avg

Import

1142ms

Disk

88MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.25.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.925 runs

installs and imports cleanly · install 0.0s · import 1.219s · 95.8MB

glibc

py 3.10–3.925 runs

installs and imports cleanly · install 4.5s · import 1.066s · 97MB

88MB installed

● package 88MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

CloudPath

✓ from cloudpathlib import CloudPath

AnyPath

✓ from cloudpathlib import AnyPath

S3Path

✓ from cloudpathlib.s3 import S3Path

GSPath

✓ from cloudpathlib.gs import GSPath

AzureBlobPath

✓ from cloudpathlib.azure import AzureBlobPath

✗ from cloudpathlib.azure import AzurePath

The specific class for Azure Blob Storage is AzureBlobPath, not a generic AzurePath.

HttpsPath

✓ from cloudpathlib.https import HttpsPath

This quickstart demonstrates how to initialize a CloudPath object for S3, write text to it, read text from it, check its existence, and then delete it. Users should ensure their cloud provider authentication environment variables are correctly set for automatic authentication. Similar patterns apply to GSPath and AzureBlobPath.

import os from cloudpathlib import CloudPath # Ensure environment variables are set for the chosen cloud provider # e.g., for S3: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY # for GS: GOOGLE_APPLICATION_CREDENTIALS (path to JSON key file) # for Azure: AZURE_STORAGE_CONNECTION_STRING # Example for S3 # Replace with your actual bucket and file name s3_file_path = CloudPath("s3://your-test-bucket/hello.txt") try: # Write to the cloud file s3_file_path.write_text("Hello from cloudpathlib!") print(f"Successfully wrote to {s3_file_path}") # Read from the cloud file content = s3_file_path.read_text() print(f"Content read: '{content}'") # Check if the file exists if s3_file_path.exists(): print(f"File {s3_file_path} exists.") # Clean up (optional) s3_file_path.unlink() print(f"File {s3_file_path} deleted.") except Exception as e: print(f"An error occurred: {e}")

Debug

Known issues

breakingThe `CloudPath.copy` method's first parameter was renamed from `destination` to `target`. Code relying on keyword arguments or positional argument names will break.

fix

Update calls to `CloudPath.copy` to use `target` as the parameter name instead of `destination`.

affects: >=0.23.0

breakingSupport for Python 3.7 has been removed. The last version compatible with Python 3.7 was v0.18.1.

fix

Upgrade your Python environment to 3.8 or newer.

affects: >=0.19.0

breakingThe `CloudPath` constructor changed how it handles a client object as the second argument. Previously, it could implicitly accept a client. Now, it needs to be passed explicitly as a keyword argument (e.g., `CloudPath('s3://...', client=my_client)`).

fix

If you were passing a client object as the second positional argument to `CloudPath` or `AnyPath`, ensure you pass it using the `client=` keyword argument.

affects: >=0.22.0

deprecatedThe environment variable `CLOUPATHLIB_FILE_CACHE_MODE` (with a typo) was deprecated and support for it has been removed. The correct environment variable is `CLOUDPATHLIB_FILE_CACHE_MODE`.

fix

Update your environment configurations to use `CLOUDPATHLIB_FILE_CACHE_MODE` instead of `CLOUPATHLIB_FILE_CACHE_MODE`.

affects: >=0.21.0 (removed), >=0.19.0 (deprecated)

gotchaThe default for `missing_ok` in `CloudPath.unlink()` is `True`, unlike `pathlib.Path.unlink()` where it defaults to `False`. This means `unlink()` will not raise an error if the file does not exist by default.

fix

If you require an error to be raised when attempting to unlink a non-existent file, explicitly pass `missing_ok=False` to `unlink()`.

affects: All versions

gotcha`CloudPath.rmdir()` will raise a `DirectoryNotEmptyError` if the directory is not empty. To recursively remove a non-empty directory, you must use `CloudPath.rmtree()`.

fix

Use `path.rmtree()` for recursive directory deletion, or ensure directories are empty before calling `path.rmdir()`.

affects: All versions

gotchaAn `ImportError` due to an incompatible `google-cloud-storage` version was fixed in v0.18.1 by not using `transfer_manager` if unavailable. Users on older `google-cloud-storage` versions might encounter this.

fix

Upgrade to `cloudpathlib` v0.18.1 or newer, and ensure your `google-cloud-storage` dependency is compatible or also updated.

affects: <0.18.1

gotchaWhen performing operations on cloud paths, `cloudpathlib` requires appropriate authentication credentials for the target cloud provider (e.g., S3, GCS, Azure). Without these, operations will fail with 'Unable to locate credentials' or similar authentication errors.

fix

Ensure your environment or system is configured with valid credentials for the respective cloud provider. For AWS S3, set `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN` (if applicable). For Google Cloud Storage, set `GOOGLE_APPLICATION_CREDENTIALS` to the path of your service account key file. For Azure Blob Storage, configure environment variables like `AZURE_STORAGE_ACCOUNT_NAME` and `AZURE_STORAGE_ACCOUNT_KEY` or `AZURE_STORAGE_CONNECTION_STRING`.

affects: All versions

gotchaOperations with cloud providers (S3, GCS, Azure) will fail with 'Unable to locate credentials' if environment variables, configuration files, or other authentication methods are not correctly set up for the respective cloud client library.

fix

Ensure your environment is configured with appropriate credentials for your cloud provider (e.g., AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION for S3; GOOGLE_APPLICATION_CREDENTIALS for GCS; AZURE_STORAGE_ACCOUNT_NAME, AZURE_STORAGE_ACCOUNT_KEY for Azure).

affects: All versions

Errors

Common errors & fixes

cloudpathlib.exceptions.MissingDependenciesError: To use S3Path, you must install cloudpathlib with the 's3' extra: pip install "cloudpathlib[s3]"

You are trying to use a cloud-specific Path class (e.g., `S3Path`, `GSPath`, `AzureBlobPath`) or `CloudPath` with a cloud URI scheme, but the necessary cloud provider SDK dependencies (like `boto3` for S3, `google-cloud-storage` for GCS, or `azure-storage-blob` for Azure) were not installed along with `cloudpathlib` as optional 'extras'.

fix

Install `cloudpathlib` with the appropriate extras for your cloud provider. For S3, use `pip install "cloudpathlib[s3]"`. For Google Cloud Storage, use `pip install "cloudpathlib[gs]"`. For Azure Blob Storage, use `pip install "cloudpathlib[azure]"`. To install all, use `pip install "cloudpathlib[all]"`.

botocore.exceptions.NoCredentialsError: Unable to locate credentials.

You are attempting to access a cloud storage resource (e.g., S3, GCS, Azure Blob Storage) using `cloudpathlib` without configuring the necessary authentication credentials for the respective cloud service.

fix

Configure your cloud credentials. For AWS S3, set `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables or use an `AWS_PROFILE`. For public S3 buckets, instantiate `S3Client` with `no_sign_request=True` and use that client. For Google Cloud Storage, set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable. For Azure, set `AZURE_STORAGE_CONNECTION_STRING`. Alternatively, explicitly instantiate and configure a client object (e.g., `S3Client`, `GSClient`, `AzureBlobClient`) with your credentials.

OSError: Cannot save file into a non-existent directory: '/var/folders/hf/cnzhkc851mqcmg0f8nc41z6h0000gn/T/tmp67yk4w9b/human-datalake/projects/raw/folder'

When using `cloudpathlib` objects with external libraries (like Pandas' `to_csv` or `to_parquet`) that rely on `os.fspath()` or expect a local file path for write operations, `cloudpathlib`'s `__fspath__` method returns the path to its local cache. If the parent directories for this local cache path do not exist, the external library will raise an `OSError`. Additionally, writing to this local cache directly will not upload the file to the cloud.

fix

Instead of directly passing a `CloudPath` object to external libraries for writing, explicitly open the `CloudPath` object for writing and pass the resulting file-like object, or write to a temporary local file first and then upload it using `cloudpathlib`'s `upload_from` method. For example, for Pandas: `with my_cloud_path.open('w') as f: df.to_csv(f)` (or `to_parquet`, etc.) or `df.to_csv('local_temp_file.csv'); my_cloud_path.upload_from('local_temp_file.csv')`.

ModuleNotFoundError: No module named 'pathlib._local'

This error occurs when `cloudpathlib` is used with Python 3.13 or 3.14 (or potentially newer versions) due to internal changes in Python's `pathlib` module, where private APIs that `cloudpathlib` previously relied on have been moved or removed.

fix

Upgrade `cloudpathlib` to a version that officially supports Python 3.13+ (e.g., v0.20.0 or later, as mentioned in a GitHub issue for 3.13 support). If a newer version of `cloudpathlib` is not yet released that supports your specific Python version, you may need to either downgrade your Python version or wait for an updated `cloudpathlib` release. Check the `cloudpathlib` GitHub repository or PyPI page for the latest compatibility information.

Upgrade

Version history

0.25.0latest on PyPI · released Aug 22, 2026

Audit

Dependencies

typing-extensionsrequiredRequired for type hints and compatibility.

boto3optionalOptional, required for AWS S3 integration.

google-cloud-storageoptionalOptional, required for Google Cloud Storage integration.

azure-storage-bloboptionalOptional, required for Azure Blob Storage integration.

azure-storage-file-datalakeoptionalOptional, required for Azure Data Lake Storage Gen2 integration.

Agent activity

49 hits · last 30 days