AWS Serverless Scaling Considerations for SQS

Scaling Considerations for AWS SQS

The SQS queue buffers API Gateway queries, allowing for asynchronous processing.

As a Lambda event source, AWS SQS controls polling the queue for you.

But you still have control over other performance-related settings.

W3schools.com collaborates with Amazon Web Services to deliver digital training content to our students.

Lambda automatically uses five parallel processes to clear the queue.

These five simultaneous processes indicate Lambda is calling your Lambda function five times.

To prevent having your Lambda function throttled, make sure the reserved concurrency is at least five.

If the Lambda service detects an increase in queue size, it raises batch size.

That implies it will call more Lambda functions concurrently.

Lambda will keep adding processes per minute until the queue slows or hits capacity.

Unless otherwise specified, the maximum concurrency is 1000.

The Lambda service will reduce the number of processes polling the queue on Lambda function failures.

Failures are placed in the dead-letter queue and deleted from your SQS queue.

If you choose a batch size that is too big, your Lambda function may timeout.

If it happens frequently, Lambda slows polling, causing a queue blockage.

Unprocessed messages become visible if the visibility timeout expires before the Lambda function processes them.

Those messages will need to be processed again, increasing queue size.

This raises the possibility of duplicate message processing.

The ideal approach is to set your visibility timeout to 6 times the function timeout.

It is critical to set the "MaxReceive" count to balance for high concurrency.

It helps with keeping things moving in the queue when sending too many messages to the dead-letter queue.

The table below shows the characteristics of an SQS queue as a Lambda event source.

Parameter	Value/Limit	How the parameter is set
Batch message size limit	1-10	Configured with event source on the Lambda function
Number of default pollers	5	Managed by the Lambda service
Lambda's rate of increasing parallel pollers	Up to 60 per minute	Managed by the Lambda service
Number of concurrent batches Lambda handles	Up to 10000	Managed by the Lambda service
Number of concurrent Lambda functions	The lesser of 1,000, function or account limit	Reserved concurrency limit
Messages per queue	No limit	N/A
Visibility timeout	0 seconds to 12 hours	Configured on the queue
Number of retries	1-1000	Maxreceive count
Function timeout	0 seconds to 15 minutes	Configured on the function