Azure Event Hubs Documentation

Scaling Azure Event Hubs

Effectively scaling Azure Event Hubs is crucial for handling fluctuating data ingestion and processing loads. Understanding the mechanisms and strategies for scaling ensures your application remains performant and cost-effective.

Understanding Scaling Factors

Azure Event Hubs scaling is primarily driven by two key resources:

• Throughput Units (TUs): These define the ingress and egress capacity of an Event Hub namespace. Each TU provides a fixed amount of ingress and egress bandwidth and connections.
• Partitions: Partitions are fundamental to Event Hubs' parallelism. The number of partitions determines the maximum concurrent readers and the degree of parallelism for message processing.

Scaling Up vs. Scaling Out

You can scale Event Hubs in two primary ways:

• Scaling Up: This involves increasing the number of Throughput Units (TUs) allocated to your Event Hub namespace. This directly increases the overall ingress and egress capacity of the namespace, allowing you to handle more data throughput.
• Scaling Out: This refers to increasing the number of partitions for a specific Event Hub. This enhances parallelism by allowing more concurrent consumers to read from the hub. It's important to note that the number of partitions is a fixed setting once an Event Hub is created and cannot be changed. If you need more partitions, you must create a new Event Hub with the desired number.

Strategies for Scaling

1. Manual Scaling (TUs)

You can manually adjust the number of TUs through the Azure portal, Azure CLI, or SDKs.

Note: While manual scaling offers direct control, it requires monitoring and proactive adjustments based on anticipated traffic patterns. Scaling up or down can take a few minutes to take effect.

2. Auto-Inflate

Auto-Inflate is a feature that automatically scales the number of TUs upward as the load increases, up to a configured maximum. This is a more dynamic approach to managing throughput.

• How it works: When the ingress or egress traffic exceeds the current TU capacity, Auto-Inflate incrementally increases the TUs.
• Configuration: You set a minimum and maximum number of TUs.

# Example Azure CLI command (illustrative)
az eventhubs namespace update \
    --resource-group MyResourceGroup \
    --name MyEventHubNamespace \
    --enable-auto-inflate true \
    --maximum-throughput-units 20
            

3. Scaling Partitions

The number of partitions should be determined based on your expected maximum number of concurrent consumers and the desired throughput per partition. A common best practice is to choose a partition count that is a power of two.

Warning: Once an Event Hub is created, the number of partitions cannot be changed. If you need to change the number of partitions, you must create a new Event Hub.

Consider these factors when choosing the number of partitions:

• Parallelism: If you have 10 consumers that need to process messages in parallel, you should ideally have at least 10 partitions.
• Throughput: Each partition has its own ingress and egress limits. Ensure that your total TU capacity is sufficient to support the traffic distributed across all partitions.
• Consumer Group Load Balancing: Event Hubs distributes partitions evenly among consumers in a consumer group. More partitions can lead to better load distribution if you have many consumers.

Monitoring and Performance Tuning

Continuous monitoring is essential for effective scaling.

• Azure Monitor Metrics: Key metrics to watch include:
  • Incoming Requests and Outgoing Requests: Monitor for throttling (HTTP 429 errors).
  • Incoming Messages/s and Outgoing Messages/s: Track overall data flow.
  • Captured Messages/s: If using Event Hubs Capture.
  • Throttled Requests: Directly indicates when limits are being hit.
• Activity Log: Monitor for any scaling operations or errors.

Choosing the Right Strategy

The optimal scaling strategy depends on your workload characteristics:

• Predictable, High Throughput: Manual scaling with carefully chosen TUs and partitions.
• Variable, Unpredictable Throughput: Auto-Inflate for TUs, combined with a well-chosen number of partitions for parallelism.
• Need for High Parallelism: Ensure your partition count aligns with your maximum concurrent consumer needs.

Regularly review your metrics and adjust your scaling strategy as your application evolves.