Registry / aws / s3torchconnectorclient

s3torchconnectorclient

library1.5.0pypypi✓ verified 83d ago

The `s3torchconnectorclient` library is an internal S3 client implementation that underpins the `s3torchconnector` library. It provides high-throughput data access and checkpointing capabilities for PyTorch training jobs interacting with Amazon S3. It is currently at version 1.5.0 and is actively developed with regular releases, often in sync with the broader `s3torchconnector` project.

aws ai-ml data

pip install s3torchconnectorclient

package ↗

INSTALL

IMPORT

SIG · S3TORCHCONNECTORCL

s3torchconnectorclient

awspythonv1.5.0

Install

1.9s avg

Import

—

Disk

28MB

Pass rate

5/ 10

Env Coverage5 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v1.5.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.920 runs

build_error

glibc

py 3.10–3.920 runs

installs and imports cleanly · install 1.9s · import 0.000s · 31MB

28MB installed

● package 28MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

S3Exception

✓ from s3torchconnectorclient import S3Exception

✗ from s3torchconnectorclient import S3ClientConfig

This quickstart demonstrates how to directly configure the `S3ClientConfig` which is part of `s3torchconnectorclient`. While direct interaction with this low-level client is possible, it's more commonly used by passing `S3ClientConfig` instances to higher-level APIs provided by the `s3torchconnector` library for datasets and readers. This example sets a custom part size and throughput target. Ensure AWS credentials are configured (e.g., via environment variables or AWS CLI) for actual S3 operations.

import os from s3torchconnectorclient import S3ClientConfig # Configure the S3 client directly. This configuration is typically # passed to higher-level constructors within the `s3torchconnector` library. config = S3ClientConfig( part_size=10 * 1024 * 1024, # Example: 10 MiB part size for transfers throughput_target_gbps=5.0, # Example: Target 5 Gbps throughput profile=os.environ.get('AWS_PROFILE', None) # Use an AWS profile if specified ) print(f"S3ClientConfig created with part size: {config.part_size / (1024*1024):.1f} MiB") print(f"S3ClientConfig created with throughput target: {config.throughput_target_gbps} Gbps") print(f"S3ClientConfig using AWS profile: {config.profile}") # In a real application, 'config' would typically be used like this (requires 's3torchconnector'): # from s3torchconnector import S3ReaderConstructor # reader_constructor = S3ReaderConstructor.sequential(s3_client_config=config) # dataset = S3MapDataset.from_prefix(DATASET_URI, region=REGION, reader_constructor=reader_constructor)

Debug

Known issues

breakingIn version 1.5.0, the internal S3Client now returns `HeadObjectResult` instead of `ObjectInfo`. `HeadObjectResult` does not include the `key` field, which might break custom reader implementations that directly relied on the `key` field from `ObjectInfo`.

fix

Update custom S3 client logic to use `HeadObjectResult` and adapt if the `key` field was previously accessed. For most users leveraging `s3torchconnector`'s high-level APIs, this change is internal and should not require direct modification.

affects: >=1.5.0

breakingStarting with version 1.5.0, `DCPOptimizedS3Reader` became the new default reader for `S3StorageReader` in `s3torchconnector`. While this offers performance improvements, it might lead to behavioral changes, especially with specific access patterns or error handling.

fix

Review the documentation for `DCPOptimizedS3Reader Errors` and test existing workloads thoroughly. If previous behavior is required, explicitly configure `S3StorageReader` with `S3ReaderConstructor.sequential()` or `S3ReaderConstructor.range_based()`.

affects: >=1.5.0

deprecatedPython 3.8 support is being deprecated and will be removed in a future release. PyTorch itself has stopped supporting Python 3.8 after v2.4.1.

fix

Upgrade to Python 3.9 or newer. The library currently supports Python 3.8-3.14.

affects: future release

deprecatedmacOS x86_64 wheel support will be deprecated in a future release.

fix

Users on macOS x86_64 should plan to migrate to an ARM-based Mac or use a Linux environment.

affects: future release

gotchaBeginning with `boto3` v1.36.0, AWS SDK for Python introduced new default integrity protections for S3 clients (checksums on Put and validation on Get). This can cause issues when interacting with third-party S3-compatible services that may not fully support these new defaults.

fix

If experiencing issues with third-party S3 services, you can disable the new integrity protections by setting `request_checksum_calculation='when_required'` and `response_checksum_validation='when_required'` in your AWS configuration (e.g., via `~/.aws/config` or environment variables). It is generally not recommended to disable these for Amazon S3 itself.

affects: boto3 >=1.36.0

gotcha`S3Reader` instances (which are part of the `s3torchconnector`'s underlying read mechanism, utilizing `s3torchconnectorclient`) are not thread-safe.

fix

Do not share `S3Reader` instances across multiple threads. For multiprocessing with `DataLoader`, each worker process automatically creates its own `S3Reader` instance, which is the recommended pattern.

affects: All

Upgrade

Version history

1.5.0latest on PyPI · released Feb 20, 2026

Audit

Dependencies

s3torchconnectorrequiredThis client is an internal component of `s3torchconnector` and is primarily used through its higher-level APIs.

boto3requiredRequired for AWS S3 interactions; specific versions may introduce behavioral changes (see warnings).

botocorerequiredUnderlying AWS SDK for Python, often installed with boto3.

Agent activity

29 hits · last 30 days

node

OpenAI (training)

Resources

packagehttps://pypi.org/project/s3torchconnectorclient/ ↗