Understanding Partitioning in Azure Event Hubs

Partitioning is a fundamental concept in Azure Event Hubs that enables scalability, parallel processing, and high throughput. An Event Hub is divided into one or more partitions. Each partition acts as an ordered, immutable sequence of events.

How Partitioning Works

When you create an Event Hub, you specify the number of partitions. This number determines the maximum degree of parallelism for consuming events from the Event Hub. Events are distributed across these partitions based on a partition key. If no partition key is provided, Event Hubs will distribute events in a round-robin fashion.

  • Ordered Sequence: Within each partition, events are strictly ordered by the time they are enqueued.
  • Independent Processing: Each partition can be processed independently by a consumer group.
  • Scalability: Increasing the number of partitions allows for increased throughput and parallel consumption.
Diagram illustrating Event Hubs partitioning

Conceptual illustration of Event Hubs partitioning.

Partition Key

The partition key is a crucial element for controlling the distribution of events. When an event is sent to an Event Hub with a partition key, Event Hubs uses a hash of the key to determine which partition the event will be written to. This ensures that all events with the same partition key are sent to the same partition.

Benefits of Using Partition Keys:

  • Ordered Processing for Related Events: By using a common key (e.g., a user ID, device ID, or session ID), you can ensure that all events related to that key are processed in the order they were sent, as they will land in the same partition.
  • Predictable Distribution: You can predict which partition an event will go to based on its key.
  • Load Balancing: While Event Hubs handles some round-robin distribution, strategic use of partition keys can help balance the load across partitions.

Choosing a Partition Key:

The choice of partition key depends on your application's needs. Consider the following:

  • Cardinality: A key with high cardinality (many unique values) will distribute events more evenly.
  • Event Correlation: Keys that represent logical groupings of events are ideal for ordered processing.
  • Consumer Affinity: If you need specific consumers to process specific sets of events, partition keys can help achieve this.

Partition ID

Each partition is identified by a unique Partition ID, which is a zero-based integer (e.g., 0, 1, 2, ...). When receiving messages, you can optionally specify a partition ID to read from a specific partition.

Working with Partitions in Code

When sending messages, you can specify the partition key. For example, using the .NET SDK:


using Azure.Messaging.EventHubs;
using Azure.Messaging.EventHubs.Producer;
using System.Threading.Tasks;

string connectionString = "YOUR_EVENTHUB_CONNECTION_STRING";
string eventHubName = "YOUR_EVENTHUB_NAME";

await using var producerClient = new EventHubProducerClient(connectionString, eventHubName);

var batchOptions = new EventHubsBatchOptions
{
    PartitionKey = "user-123" // Example partition key
};

using EventDataBatch eventBatch = await producerClient.CreateBatchAsync(batchOptions);

if (!eventBatch.TryAddMessage(new EventData(BinaryData.FromString("Event 1 for user-123"))))
{
    // The batch is full, send it and create a new one
    await producerClient.SendAsync(eventBatch);
    // ... create new batch and add message ...
}
// ... add more messages ...

await producerClient.SendAsync(eventBatch);

When receiving messages, consumers typically read from all partitions within a consumer group. The Event Hubs SDKs provide mechanisms to manage this, often abstracting away direct partition management for simplicity.

Key Takeaway: The number of partitions directly impacts the maximum parallelism of your Event Hub. Choose the number of partitions carefully based on your expected throughput and processing needs. Once set, the number of partitions for an Event Hub namespace cannot be changed. You can, however, create a new Event Hub with a different number of partitions.

Partition Limits

The maximum number of partitions is determined by your Event Hubs pricing tier:

  • Basic: Up to 2 partitions.
  • Standard: Up to 32 partitions (configurable at creation time). For higher scale needs, contact Azure Support.
  • Premium: Up to 1024 partitions per namespace.

Refer to the Azure Event Hubs pricing page for the most up-to-date information.