Registry / aws / amazon-sqs-extended-client

amazon-sqs-extended-client

JSON →
library1.0.1pypypi✓ verified 83d ago

The Amazon SQS Extended Client Library for Python extends Amazon SQS to handle message payloads larger than the standard 256 KB limit, supporting sizes up to 2 GB. It achieves this by transparently storing the message payload in an Amazon S3 bucket and sending a reference to the S3 object in the SQS message. The library is currently at version 1.0.1 and maintains a release cadence tied to feature development and AWS SDK updates.

pip install amazon-sqs-extended-client
INSTALL
IMPORT
SIG · AMAZON-SQS-EXTENDE
A
amazon-sqs-extended-client
awspythonv1.0.1
Install
3.6s avg
Import
Disk
49MB
Pass rate
10/ 10
Env Coverage10 / 10
glibc
3.93.13
musl
3.93.13
Install & Compatibility
Where this runs
tested against v1.0.1 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
musl
py 3.103.920 runs
installs and imports cleanly · install 0.0s · import 0.000s · 50.9MB
glibc
py 3.103.920 runs
installs and imports cleanly · install 3.6s · import 0.000s · 51MB
49MB installed
● package 49MB
Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

SQSExtendedClientSession
from sqs_extended_client import SQSExtendedClientSession
from amazon_sqs_extended_client import SQSExtendedClient

This quickstart demonstrates how to initialize the `SQSExtendedClient` and use it to send and receive messages larger than the standard SQS limit. It requires an existing SQS queue and an S3 bucket configured in your AWS environment with appropriate permissions. The `always_through_s3` flag is set to `True` to ensure all messages, regardless of size, are handled via S3, which is often desired for consistency.

import boto3 from amazon_sqs_extended_client import SQSExtendedClient import os # Configure AWS credentials and region # Ensure AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION are set as environment variables aws_region = os.environ.get('AWS_REGION', 'us-east-1') sqs_client = boto3.client('sqs', region_name=aws_region) s3_client = boto3.client('s3', region_name=aws_region) # SQS Queue URL and S3 Bucket Name must exist and be accessible # Replace with your actual SQS Queue URL and S3 Bucket Name SQS_QUEUE_URL = os.environ.get('SQS_QUEUE_URL', 'YOUR_SQS_QUEUE_URL') S3_BUCKET_NAME = os.environ.get('S3_BUCKET_NAME', 'YOUR_S3_BUCKET_NAME') # Initialize the SQS Extended Client sqs_extended_client = SQSExtendedClient( sqs_client=sqs_client, s3_client=s3_client, bucket_name=S3_BUCKET_NAME, always_through_s3=True # Always store messages in S3, regardless of size ) # Example: Send a large message large_message_body = 'A' * 300000 # 300KB message, exceeding SQS limit print(f"Sending a large message ({len(large_message_body) / 1024} KB)...") send_response = sqs_extended_client.send_message( QueueUrl=SQS_QUEUE_URL, MessageBody=large_message_body ) print(f"Message sent! Message ID: {send_response.get('MessageId')}") # Example: Receive the message print("\nReceiving messages...") receive_response = sqs_extended_client.receive_message( QueueUrl=SQS_QUEUE_URL, MaxNumberOfMessages=1 ) messages = receive_response.get('Messages', []) if messages: message = messages[0] received_body = message.get('Body') print(f"Received message body (first 100 chars): {received_body[:100]}...") print(f"Received message length: {len(received_body)} bytes") # Delete the message after processing sqs_extended_client.delete_message( QueueUrl=SQS_QUEUE_URL, ReceiptHandle=message['ReceiptHandle'] ) print("Message deleted.") else: print("No messages received.")
Debug
Known issues
gotchaThe Amazon SQS Extended Client Library for Python explicitly depends on an S3 bucket for storing large message payloads. Ensure the specified S3 bucket exists and that your AWS credentials (used by Boto3 clients) have the necessary permissions (s3:PutObject, s3:GetObject, s3:DeleteObject) for the bucket and sqs:SendMessage, sqs:ReceiveMessage, sqs:DeleteMessage for the SQS queue. Lack of S3 permissions is a common cause of `Failed to get the S3 object which contains the payload` errors.
fix
Verify IAM policies for the AWS user/role sending and receiving messages. Grant `s3:PutObject`, `s3:GetObject`, `s3:DeleteObject` on the S3 bucket and `sqs:*` on the SQS queue.
affects: All versions
gotchaThere are several similarly named Python libraries (`aws-sqs-ext-client`, `sqs-extended-client`, `pysqs-extended-client`) that aim to provide similar functionality. Ensure you are using `amazon-sqs-extended-client` (from `awslabs` on GitHub) for the official and actively maintained version to avoid compatibility issues or unexpected behavior.
fix
Always install using `pip install amazon-sqs-extended-client` and import with `from amazon_sqs_extended_client import SQSExtendedClient`.
affects: All versions
gotchaWhen deleting messages, the extended client automatically handles the deletion of the corresponding S3 object if the message payload was stored in S3. However, if the S3 object is manually deleted before the SQS message is processed and deleted, the consumer might fail to retrieve the full message body, leading to `Failed to get the S3 object which contains the payload` errors.
fix
Rely on the `SQSExtendedClient.delete_message` or `delete_message_batch` calls to ensure both the SQS message and its S3 payload are removed synchronously. Avoid manual deletion of S3 objects referenced by active SQS messages.
affects: All versions
Upgrade
Version history
1.0.1latest on PyPI · released Sep 11, 2024
Audit
Dependencies
boto3requiredThe library is built upon the AWS SDK for Python (Boto3) for interacting with SQS and S3.
Agent activity
43 hits · last 30 days
node
36
Perplexity
1
OpenAI (training)
1
Resources
amazon-sqs-extended-client — pip install amazon-sqs-extended-client · libregistry