Understanding Partitioning in Azure Event Hubs

Partitioning is a fundamental concept in Azure Event Hubs that enables high throughput and scalability. Each Event Hub is divided into one or more partitions. Partitions are ordered, immutable sequences of events. Events written to a partition are assigned a sequence number that is unique only within that partition.

Diagram illustrating Event Hubs with multiple partitions

Conceptual Diagram of an Event Hub with Partitions.

Key Aspects of Partitioning

  • Scalability: Partitions allow Event Hubs to scale horizontally. Producers can write to different partitions concurrently, and consumers can read from different partitions in parallel.
  • Ordering: Within a single partition, events are guaranteed to be ordered. This is crucial for applications that require strict event ordering for specific entities or streams.
  • Throughput: By distributing events across multiple partitions, Event Hubs can achieve higher ingest and egress rates.
  • Consumer Groups: Consumers access Event Hubs through consumer groups. Each consumer group maintains its own offset within each partition, allowing multiple applications or instances of the same application to consume events independently.

Partition Keys

When sending events to an Event Hub, you can optionally specify a partition key. The partition key is a string that is used to determine which partition an event is sent to. Event Hubs uses a hash of the partition key to determine the target partition. This ensures that all events with the same partition key are sent to the same partition, maintaining event order for that key.

If no partition key is provided, Event Hubs will use a round-robin approach to distribute events across available partitions. This is suitable for scenarios where ordering per entity is not critical, and maximizing throughput is the primary goal.

When to Use Partition Keys:

  • When strict ordering of events related to a specific entity (e.g., a user ID, device ID, or order ID) is required.
  • To ensure that related events are processed together by a single consumer instance within a consumer group.

Choosing a Partition Key:

A good partition key should:

  • Be consistent for related events.
  • Distribute events evenly across partitions to avoid hot partitions.
  • Have a high cardinality if even distribution is desired.

Partition Assignment

The number of partitions in an Event Hub is determined at creation time and can be updated later (though this operation has implications and should be planned carefully). The maximum number of partitions is limited by the chosen tier and capacity settings.

Consumers within a consumer group coordinate to read from partitions. If you have N partitions and M consumer instances within a consumer group, each consumer instance will typically be responsible for reading from a subset of the partitions. This parallel processing is key to Event Hubs' scalability.

Tip: Avoid choosing partition keys that result in "hot" partitions, where one partition receives a disproportionately large number of events. This can lead to performance bottlenecks and impact overall throughput.

Example: Sending Events with a Partition Key (Conceptual)

Here's a conceptual representation of sending events using a partition key:


// Assuming you have an EventHubProducerClient configured

var eventData1 = new EventData("Sensor reading for device 123");
eventData1.Properties["PartitionKey"] = "device-123"; // Assign partition key

var eventData2 = new EventData("User login event for user 456");
eventData2.Properties["PartitionKey"] = "user-456"; // Assign partition key

var eventData3 = new EventData("Another reading for device 123");
eventData3.Properties["PartitionKey"] = "device-123"; // Will go to the same partition as eventData1

await producerClient.SendAsync(new List<EventData> { eventData1, eventData2, eventData3 });

In this example, eventData1 and eventData3, both associated with "device-123", will be sent to the same partition, ensuring their order is preserved. eventData2, with a different key, will go to a different partition.

Monitoring Partitions

It's important to monitor your Event Hub partitions for:

  • Ingress/Egress Throughput: Ensure even distribution across partitions.
  • Latency: Identify any partitions experiencing higher than expected latency.
  • Throttling: Check if producers or consumers are being throttled due to partition limitations.

Azure Monitor provides metrics and tools to help you track the health and performance of your Event Hub partitions.

Further Reading