Azure Storage Documentation

Azure Storage Queues Best Practices

This document outlines best practices for using Azure Storage Queues effectively, ensuring performance, scalability, and reliability for your message-driven applications.

1. Design for Idempotency

Queue processing should be idempotent. This means that processing the same message multiple times should have the same effect as processing it once. This is crucial because messages can occasionally be delivered more than once due to network issues or processing failures.

• Use unique identifiers for operations.
• Implement check-and-set logic for state changes.

2. Optimize Message Size

Azure Storage Queues have a message size limit of 64 KB. While this is generally sufficient, sending very large messages can impact performance and increase costs. Consider storing large data in Blob Storage and sending a reference (URI) to the blob in the queue message.

{
  "blobUri": "https://myaccount.blob.core.windows.net/mycontainer/large-data.json",
  "operation": "process_data"
}

3. Manage Visibility Timeout Correctly

When a message is retrieved using Get Messages, it becomes invisible to other consumers for a specified duration (visibility timeout). Configure this timeout appropriately:

• Too short: Messages might be re-processed unnecessarily if consumers are slow.
• Too long: If a consumer fails, messages remain locked for an extended period, reducing availability.

Use Update Message to extend the visibility timeout if processing takes longer than expected.

4. Handle Poison Messages

A poison message is one that repeatedly causes a consumer to fail. Implement a strategy to handle these:

• Retry count: Track the number of times a message has been dequeued. If it exceeds a threshold, move it to a dead-letter queue or log it for inspection.
• `maxDequeueCount` property: Configure this on the queue to automatically move messages to the dead-letter queue after a specified number of failures.

When processing, explicitly decrement the dequeue count if the message processing is successful. Otherwise, let the visibility timeout expire for retries.

5. Scale Consumers Horizontally

Azure Storage Queues are designed for high throughput. To handle large message volumes, scale your message consumers horizontally. Run multiple instances of your worker roles or containers that process messages concurrently.

6. Monitor Queue Health

Regularly monitor key queue metrics such as:

• ApproximateMessageCount: Indicates the number of messages waiting to be processed. A consistently high count might suggest consumers are not keeping up.
• IncomingRequests and OutgoingRequests: To understand traffic patterns.
• IngressSize and EgressSize: For data transfer monitoring.

Set up alerts for critical thresholds.

7. Consider Message TTL (Time-to-Live)

Messages in Azure Storage Queues have a Time-to-Live (TTL) property. After this duration, messages are automatically deleted. Set this to an appropriate value to prevent stale messages from accumulating indefinitely.

// Example using Azure SDK
await queueClient.sendMessage("Hello world", {
    messageTimeToLive: TimeSpan.FromDays(7)
});

8. Use SAS Tokens for Granular Access Control

For scenarios where you need to grant temporary or restricted access to queues (e.g., for a specific client application), use Shared Access Signature (SAS) tokens instead of account keys.

9. Choose the Right Dequeue Strategy

When retrieving messages, consider the trade-offs:

• Single message retrieval: Simpler for processing one message at a time.
• Batch retrieval (e.g., Get Messages with a count): More efficient for high-volume processing, reducing latency and cost. Ensure your consumer can handle multiple messages concurrently.