Tutorials References Exercises Videos Menu
Create Website Get Certified Upgrade

AWS Serverless Error Handling for Stream-Based Events


Error Handling for Stream-Based Events

Stream-based event sources must keep record order consistent across shards.

Stream-based event sources could be Kinesis Data Streams or DynamoDB Streams.

If Lambda encounters an issue while processing a batch of data, it will stop processing new data.

A batch of data is a large amount of data.

To discover blocked shards, you can use the Iterator-Age metric.

It measures the age of the stream records your function recently processed.


Failure Management

By using four configuration options, you can manage failures better:

  • Bisect batch on function error
  • Maximum retry attempts
  • Maximum record age
  • On-failure destination

If a function error occurs, Lambda will split the batch in half and resume each half separately.

Maximum retry attempts and maximum record age limit the number of retries on a failed batch.

An on-failure destination allows you to send failed records for offline processing.


Error Handling For Stream-Based Events Video

W3schools.com collaborates with Amazon Web Services to deliver digital training content to our students.


How Error Handling for Stream-Based Events Works

When a function produces an error, Lambda divides the batch in half (Bisect batch).

Lambda delivers each half to your function individually, keeping record order.

When Lambda divides a batch, it also resets the retry and max-age parameters.

Lambda will keep dividing the batches until it finds the bad record.

Lambda will keep trying to send the bad record according to the maximum retry attempts.

Lambda will send it to the SNS topic defined for the on-failure destination if it continues to fail.

After removing the erroring record, Lambda goes back through each of the smaller batches it made.

The downside of this process is that some records could be processed multiple times.

It is also called idempotency.

Idempotency must be managed by you.


AWS Serverless Exercises

Test Yourself With Exercises

Exercise:

What is idempotency?

Processing same  multiple times

Start the Exercise