Registry / aws / pathy

pathy

JSON →
library0.13.0pypypi✓ verified 49d ago

Pathy extends Python's `pathlib.Path` to provide a unified, pathlib-compatible interface for interacting with local filesystems and various cloud storage services (S3, GCS, Azure Blob Storage). It aims to make file operations cloud-agnostic and seamless. The current stable version is 0.13.0, with frequent minor and patch updates.

awsgcpazureserialization
pip install pathy
Install & Compatibility
Where this runs
tested against v0.14.2 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
musl
py 3.103.9100 runs
installs and imports cleanly · install 0.0s · import 1.306s · 53.4MB
glibc
py 3.103.9100 runs
installs and imports cleanly · install 4.1s · import 1.146s · 54MB
55MB installed
● package 55MB
Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

Pathy
from pathy import Pathy
BlobStat
from pathy import BlobStat
Returned by .stat() methods for consistent metadata across local and cloud paths.

This example demonstrates basic file operations using Pathy for both local and cloud (S3) paths. For cloud operations, ensure the relevant `pathy` extras are installed (e.g., `pathy[s3]`) and environment variables for authentication are configured.

from pathy import Pathy import os # --- Local filesystem operations --- local_file_path = Pathy("./temp_pathy_local_file.txt") try: local_file_path.write_text("Hello from Pathy on local filesystem!") print(f"Local file content: '{local_file_path.read_text()}'") print(f"Local file size: {local_file_path.stat().st_size} bytes") finally: if local_file_path.exists(): local_file_path.unlink() # Clean up # --- Cloud storage operations (example: S3) --- # Replace with your bucket name and ensure AWS credentials are set as environment variables s3_bucket = os.environ.get("PATHY_TEST_S3_BUCKET", "my-pathy-test-bucket") s3_path = Pathy(f"s3://{s3_bucket}/example_dir/cloud_data.txt") # Check if AWS credentials are likely available if os.environ.get("AWS_ACCESS_KEY_ID") and os.environ.get("AWS_SECRET_ACCESS_KEY"): print(f"\nAttempting S3 operations on: {s3_path}") try: # Create parent directories (no-op if they exist for cloud paths) s3_path.parent.mkdir(parents=True, exist_ok=True) s3_path.write_text("Hello from Pathy on S3!") print(f"S3 file content: '{s3_path.read_text()}'") print(f"S3 file size: {s3_path.stat().st_size} bytes") except Exception as e: print(f"Failed to perform S3 operations. Ensure 'pathy[s3]' is installed and credentials are correct. Error: {e}") finally: if s3_path.exists(): s3_path.unlink() # Clean up print(f"Cleaned up S3 file: {s3_path}") else: print("\nSkipping S3 example: AWS_ACCESS_KEY_ID and/or AWS_SECRET_ACCESS_KEY environment variables not set.") print("Install 'pathy[s3]' and configure AWS credentials to run this example.")
Debug
Known issues
breakingPathy no longer inherits directly from `pathlib.Path`. This means some methods or properties specific to `pathlib.Path` might behave differently or be unavailable, and direct `isinstance(pathy_obj, pathlib.Path)` checks will fail.
fix
Review existing code for direct `pathlib.Path` inheritance assumptions or specific `pathlib` method calls. Pathy aims to maintain compatibility with `pathlib`'s public API, but internal structure has changed.
affects: >=0.11.0
breakingThe `Pathy.key` property now returns a `str` instead of a `Pathy` instance. This changes the type of the key used to represent the path's identifier within a bucket or root.
fix
If you were chaining `.key` operations or expecting a `Pathy` object, update your code to handle a `str` return type. Convert back to `Pathy(key_string)` if a Pathy instance is needed.
affects: >=0.11.0
breakingThe `Pathy.stat()` method now consistently returns a `pathy.BlobStat` object for both local and remote paths. Previously, local paths would return an `os.stat_result`.
fix
Update code that inspects `stat()` results to expect `BlobStat` attributes (e.g., `st_size`, `st_mtime`) which are compatible with `os.stat_result` but now provide a unified interface. Custom attribute access might need adjustment.
affects: >=0.10.0
breakingDropped support for Python 3.7. Pathy now requires Python 3.10 or newer.
fix
Upgrade your Python environment to 3.10 or later to use recent Pathy versions.
affects: >=0.11.0
breakingPathy's internal architecture, relying on `pathlib_abc`, underwent significant upgrades (0.3.x to 0.5.x). While primarily internal, users extending Pathy or relying on deep internal interfaces might experience breaking changes.
fix
If you have custom Pathy backends or complex extensions, review the `pathlib_abc` changes or the Pathy source for potential incompatibilities in abstract methods or internal types.
affects: >=0.13.0
gotchaCloud provider support requires specific 'extras' to be installed (e.g., `pathy[s3]`, `pathy[gcs]`, `pathy[azure]`). Without these, attempts to access cloud paths will result in import errors or `ClientError`.
fix
Ensure you install Pathy with the correct extras for the cloud storage you intend to use. Example: `pip install "pathy[s3]"`.
affects: All versions
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'pathy'
The 'pathy' library is not installed in your current Python environment, or the Python interpreter cannot find it in its search path.
fix
Install the pathy library using pip: `pip install pathy`
AttributeError: 'Pathy' object has no attribute 'some_method'
You are attempting to access a method or attribute on a `pathy.Pathy` object that either does not exist, is misspelled, or is specific to `pathlib.Path` and not yet implemented or applicable for cloud paths within Pathy.
fix
Check the pathy documentation for the correct method name and its availability for the specific cloud storage backend you are using. If the method is from `pathlib.Path`, ensure it's supported by `pathy.Pathy` or adjust your logic accordingly.
InvalidAuthentication: The provided authentication header is invalid.
Pathy or the underlying cloud storage SDK (e.g., for GCS, S3, or Azure Blob Storage) failed to authenticate your request, often due to incorrect, expired, or missing credentials/access keys, or insufficient permissions.
fix
Verify that your cloud provider credentials (e.g., service account key file for GCS, AWS access key ID and secret access key for S3, Azure storage account name and key) are correctly configured and have the necessary permissions for the requested operation. Check environment variables or configuration files Pathy uses for credentials.
FileNotFoundError: No such file or directory: 'gs://your-bucket/nonexistent-file.txt'
The specified cloud file or directory path does not exist in the target cloud storage bucket, or there's a typo in the path.
fix
Double-check the cloud path for correctness, including bucket name, prefixes, and object name. Ensure the file or directory actually exists in your cloud storage. You can use methods like `.exists()` on the `Pathy` object to programmatically check for existence before attempting operations.
Upgrade
Version history
0.14.2latest on PyPI
Audit
Dependencies
smart-openrequiredUsed for efficient reading/writing to remote filesystems.
boto3optionalRequired for AWS S3 backend.
google-cloud-storageoptionalRequired for Google Cloud Storage backend.
azure-storage-bloboptionalRequired for Azure Blob Storage backend.
Agent activity
66 hits · last 30 days
bytedance
5
node
4
seranking-bot
4
ahrefsbot
3
petalbot
1
Resources