s3-concat

library0.3.0pypypi✓ verified 83d ago

s3-concat is a Python library for concatenating files stored in AWS S3 without downloading them locally. It uses the S3 multipart upload API to efficiently combine objects. Current version is 0.3.0, requires Python >=3.8, and is released on PyPI. The library is in active development but has infrequent releases.

devops storage

pip install s3-concat

github ↗homepage ↗

INSTALL

IMPORT

SIG · S3-CONCAT

s3-concat

devopspythonv0.3.0

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

S3Concat

✓ from s3_concat import S3Concat

✗ from s3-concat import S3Concat

s3-concat is not a valid Python identifier; use underscores.

Concatenates multiple S3 objects into one using multipart upload.

import os from s3_concat import S3Concat s3 = S3Concat( aws_access_key_id=os.environ.get('AWS_ACCESS_KEY_ID', ''), aws_secret_access_key=os.environ.get('AWS_SECRET_ACCESS_KEY', ''), region_name=os.environ.get('AWS_REGION', 'us-east-1') ) source_bucket = 'my-source-bucket' source_keys = ['part1.csv', 'part2.csv', 'part3.csv'] dest_bucket = 'my-dest-bucket' dest_key = 'concat.csv' s3.concat(source_bucket, source_keys, dest_bucket, dest_key) print(f'Successfully concatenated files to s3://{dest_bucket}/{dest_key}')

Debug

Known issues

gotchaThe source files must be in the same bucket and region. Cross-bucket concatenation is not supported.

fix

Ensure all source files are in the same bucket, or copy them to the same bucket first.

affects: all

deprecatedParameter 'aws_access_key_id' and 'aws_secret_access_key' are deprecated; use boto3 default session or pass credentials via boto3 session object.

fix

Use a boto3.Session object and pass it to S3Concat with explicit_creds=False (or rely on default credentials).

affects: >=0.3.0

gotchaThe library does not sort source keys; concatenation order follows the list order. Ensure keys are in desired order.

fix

Sort source_keys list before calling s3.concat().

affects: all

gotchaS3 multipart upload requires each part to be at least 5MB (except the last part). If source files are smaller, they will be combined into larger parts but may still cause issues if total upload is small.

fix

Ensure total size of concatenated file is at least 5MB, or be aware that very small concatenations may fail with 'EntityTooSmall' error.

affects: all

Errors

Common errors & fixes

ModuleNotFoundError: No module named 's3_concat'

Attempted to import with hyphens instead of underscores.

fix

Use import s3_concat (underscores) instead of s3-concat.

botocore.exceptions.ClientError: An error occurred (InvalidPart) when calling the CompleteMultipartUpload operation: One or more of the specified parts could not be found

The multipart upload parts list is incorrect or parts have been deleted/overwritten.

fix

Ensure source files are not modified during the concatenation process. Also check that the bucket/keys exist and are accessible.

botocore.exceptions.ClientError: An error occurred (NoSuchKey) when calling the HeadObject operation: The specified key does not exist.

One of the source keys does not exist in the bucket.

fix

Verify that all source keys are correct and exist in the source bucket.

Upgrade

Version history

0.3.0latest on PyPI · released May 15, 2025

Audit

Dependencies

boto3requiredAWS SDK for Python, required for S3 operations

Agent activity

28 hits · last 30 days

node

Resources

githubhttps://github.com/xtream1101/s3-concat ↗

homepagehttps://github.com/xtream1101/s3-concat ↗