Azure Event Hubs - Troubleshooting

A Comprehensive Guide for Developers

Table of Contents

Common Issues and Solutions

Connection Refused/Timeout

Errors indicating the client cannot establish a connection to the Event Hubs endpoint.

  • Verify firewall rules and network security groups (NSGs) allow outbound traffic on port 5671 (AMQP) or 443 (HTTPS).
  • Check DNS resolution for the Event Hubs namespace.
  • Ensure the Event Hubs service is healthy in your region.
  • Confirm you are using the correct connection string or endpoint.

SSL/TLS Handshake Failures

Problems during the secure connection establishment.

  • Ensure your client's system time is accurate.
  • Verify that the Event Hubs endpoint certificate is trusted by your client.
  • Check for any intermediate proxies that might be intercepting SSL traffic.

Throttling Errors (e.g., 401, 403)

Client requests are being denied due to exceeding limits or insufficient permissions.

  • Review your Event Hubs tier and capacity settings.
  • Optimize your application's throughput to stay within provisioned limits.
  • Ensure your Shared Access Signature (SAS) key or Azure AD token has the correct permissions (Send, Listen, Manage).

Producer Errors

Problems encountered when sending messages to Event Hubs.

Tip: Always implement retry logic with exponential backoff for transient errors.

Message Too Large

Sending a message that exceeds the configured maximum message size.

  • Check the Event Hubs documentation for the maximum message size for your tier.
  • Split large messages into smaller ones if possible.
  • Consider compression for larger payloads.

Partitioning Issues

Unable to send to a specific partition or unexpected partitioning behavior.

  • Ensure you are not hardcoding partition IDs unless absolutely necessary.
  • Use the Event Hubs SDK's default partitioning strategy or specify a partition key for consistent routing.
  • Check if partitions are available and healthy.

Consumer Errors

Problems encountered when receiving messages from Event Hubs.

No Messages Received

Your consumer is running but not processing any messages.

  • Verify the consumer is connected to the correct Event Hub and consumer group.
  • Check if producers are actively sending messages.
  • Ensure the consumer group has not fallen too far behind; you may need to seek an earlier offset.
  • Confirm you are not using an expired SAS token.

Checkpointing Failures

Errors related to saving the consumer's progress.

  • Ensure the storage account or Azure Table configured for checkpoints is accessible and has correct permissions.
  • Check for network connectivity issues to the checkpoint storage.
  • Verify the format and structure of your checkpoint data.

Performance Bottlenecks

Slow throughput or high latency when sending or receiving messages.

High Latency

Messages taking too long to be delivered.

  • Monitor network latency between your application and Event Hubs.
  • Ensure your application instances are geographically close to the Event Hubs namespace.
  • Optimize your application's processing logic to avoid delays.
  • Consider increasing the number of partitions for higher throughput.

Low Throughput

Not achieving the expected message processing rate.

  • Check Event Hubs metrics for throttling and connection limits.
  • Scale up your Event Hubs capacity (throughput units or processing units).
  • Parallelize processing within your consumer or use multiple consumer instances.
  • Batch messages efficiently on the producer side.

Authentication & Authorization

Issues related to identity and access management.

Invalid Credentials (401 Unauthorized)

The provided credentials are not valid or have expired.

  • Double-check your connection string for typos.
  • Ensure your SAS key has not been revoked or expired.
  • If using Azure AD, verify your token is current and has the correct audience.
  • Confirm the correct key name is used in the connection string.

Insufficient Permissions (403 Forbidden)

The authenticated identity does not have the necessary rights.

  • Grant the `Microsoft.EventHub/namespaces/eventhubs/send/action` permission for producers.
  • Grant the `Microsoft.EventHub/namespaces/eventhubs/listen/action` permission for consumers.
  • Ensure you are using the correct key or role assignment.

Message Loss and Ordering

Concerns about messages not arriving or arriving out of order.

Warning: While Event Hubs guarantees ordering within a partition, cross-partition ordering is not guaranteed.

Potential Message Loss

Messages are being sent but not received.

  • Implement robust error handling and retry mechanisms on the producer.
  • Use acknowledgements or a dead-lettering mechanism if your scenario requires strict guarantees.
  • Monitor consumer lag to ensure messages are being processed.

Out-of-Order Messages

Messages are arriving in an unexpected sequence.

  • Ensure messages intended to be ordered are sent to the same partition using a partition key.
  • If processing out of order is acceptable, implement logic to reorder messages based on sequence numbers or timestamps if required.

Quota and Limits

Encountering issues due to hitting service quotas.

Exceeding Throughput Units (TUs/PUs)

Your namespace is being throttled due to high traffic.

  • Monitor the `IncomingMessagesPerSecond` and `OutgoingMessagesPerSecond` metrics.
  • Scale up your Event Hubs namespace by increasing TUs or PUs.
  • Distribute load across more partitions if possible.

Connection Limits

Your application is hitting the maximum number of concurrent connections.

  • Optimize connection usage by reusing connections where possible.
  • Monitor `ConnectionsEstablished` metrics.
  • Consider using the AMQP protocol, which is more connection-efficient than HTTP for high-volume scenarios.

Logging and Monitoring

Strategies for diagnosing issues effectively.

Tip: Integrate your applications with Azure Monitor for comprehensive insights into Event Hubs performance and health.

Lack of Visibility

Difficulty understanding what's happening within Event Hubs.

  • Enable diagnostic logs for your Event Hubs namespace.
  • Set up alerts in Azure Monitor for key metrics like `ServerErrors`, `ThrottledRequests`, and `ConnectionDropCount`.
  • Instrument your producer and consumer applications with detailed logging, including correlation IDs for tracing requests.

Troubleshooting Logs

Interpreting error messages and logs.

Common log entries to look for:

  • HTTP status codes (e.g., 401, 403, 429).
  • AMQP error conditions (e.g., `amqp:unauthorized`, `amqp:link:transfer-limit-exceeded`).
  • SDK-specific error codes and messages.

Example of a producer log entry:

INFO [Producer] Sending message... { correlationId: "abc-123", partitionKey: "key1" }

Example of an error log entry:

ERROR [Consumer] Failed to receive messages: "amqp:link:transfer-limit-exceeded" { consumerGroup: "$Default", namespace: "my-eventhub.servicebus.windows.net" }