Amazon SQS or Simple Queue service is a distributed messaging queue that helps in Producer-consumer connectivity for distributed systems. You can read more about SQS here:

In SQS when a message is pushed to the queue it remains in the queue until consumed by the consumer.
When a consumer retrieves a message from an SQS queue, the message is not immediately deleted from the queue. Instead, it enters a “hidden” state for a specified period of time, during which it is not visible to other consumers.

This hidden period of time is Visibility Timeout.

So, How do we delete a message once it is consumed? and How do we retry if it fails? We will discuss everything about how Visibility Timeout ensures the integrity and reliability of message processing in distributed systems.

Let’s have a look

First, we will discuss the states a message goes through during its lifetime.

States Of message:

Here are three basic states of a message:

1. Message produced to a queue: When a producer produces a message to the queue it gets stored in the queue.

2. Received by a consumer: When a consumer polls a message from a queue it goes to In Flight state.

3. Message Deleted: When a consumer completes processing the message and it deletes the message from the queue.

In-Flight State:

A message is considered to be In-Flight after a consumer receives it from the queue till the message is deleted from the queue. That’s between states 2 and 3.

There can be a maximum of 120,000 messages that can be In-Flight for Standard Queues and 20,000 for FIFO Queues. You can check these Quotas in AWS Service Quotas.

Note:*These quotas only apply to messages that are In-Flight i.e received by the consumer and yet to be deleted.*

Let’s now discuss how Visibility Timeout comes into the picture.

Visibility Timeout:

When a message is received by the consumer it becomes invisible to other consumers. We do it for these three reasons:

• Since SQS is a distributed system we can not guarantee that a message is received by the consumer. There can be a connectivity issue or consumer applications might fail in between. So, to prevent loss of message instead of deleting we hide it.

• We prevent concurrency of the process. Since it is a distributed system there will be more than one consumer so to prevent a message from being received by multiple consumers we make it invisible as soon as one consumer receives the message.

• To give consumer time for processing messages successfully.

How long does the message remain hidden?

By default, the Visibility timeout for a queue is 30 seconds. We can configure it to a max of 12 hours. we can change this by:

• Changing the timeout in queue configs.

• Changing inside the consumer when it receives the message making ChangeMessageVisibility call.

Visibility Timeout starts when a message goes into In-flight or received by the consumer.

How do we delete or retry the message?

When a message is received by the consumer it has a Receipt HandlereceiptHandle parameter inside. The Receipt Handle gets generated every time a message is received by the consumer thus, it is always unique.

We usereceiptHandle to change message visibility timeout inside consumers.

• Deleting the message: Once the consumer has completed processing the message we can use the receiptHandle to delete the message. Deleting a message is very important since we have a limit on our In-Flight messages as discussed above.

• Message Retry: If in case our consumer fails to process the message we can throw an error and the message will become visible again after Visibility Timeout is over.
  Also, if the consumer fails to send DeleteRequest after the message is successfully processed or the consumer timeout or it does not send any response, the message will again become visible for other consumers after the Visibility Timeout Expires.

This is how Visibility Timeout provides reliability and protects the integrity of messages in a distributed system.

Bonus:

Important points to remember:

1. Visibility Timeout and In-flight start when a message is received from the queue.

2. Deleting a message is important after it has been successfully processed cause of the limit on In-Flight messages.

3. Visibility Timeout can go up to 12 hours from the time receive call is made to the queue not when the message becomes available to the consumer. So changing the timeout in the consumer should consider the latency poll operation.

4. Messages are retried after Visibility Timeout even if the consumer throws an exception.

5. You can terminate the Visibility Timeout using the call ChangeMessageVisibility and set the parameter VisibilityTimeout to 0 seconds.

That’s all about Visibility Timeout in SQS. Send us your feedback using the message button below. Your feedback helps us create better content for you and others. Thanks for reading!

If you like the article please like and share. To show your support!
Thanks.