Consumer Groups

Consumer groups are a core concept in Azure Event Hubs that enable multiple applications or services to independently consume a stream of events from an Event Hub. Each consumer group maintains its own read position within the event stream.

Why Use Consumer Groups?

Without consumer groups, only one application could read from an Event Hub at a time. Consumer groups unlock several key scenarios:

Multiple Consumers: Different applications can process the same event data without interfering with each other. For example, one application might archive events to Blob Storage, while another processes them for real-time analytics.
Independent Processing: Each consumer group can read events at its own pace and restart consumption from its last known position without affecting other groups.
Fault Tolerance and Scalability: If one consumer in a group fails, others can continue processing. You can also scale out consumers within a group to handle higher throughput.
Development and Testing: New applications or services can be developed and tested against the same event stream without impacting existing production consumers.

How Consumer Groups Work

When you create an Event Hub, a default consumer group named $Default is automatically created. You can create additional consumer groups as needed. Each consumer group is identified by a unique name.

When a consumer client connects to an Event Hub, it specifies which consumer group it belongs to. The Event Hubs service then ensures that:

Each consumer within a specific consumer group reads distinct events from the partitions.
The read offset (or sequence number) is maintained independently for each consumer group.

Key Characteristics

Unique Names: Consumer groups must have unique names within an Event Hub.
Independent State: Each consumer group tracks its own offset for each partition.
Shared Data: All consumer groups read from the same underlying event stream.
Partition Association: Consumers within a group are distributed across the Event Hub's partitions. The Event Hubs SDK handles load balancing and ensuring each consumer gets a unique set of partitions to process.

Creating and Managing Consumer Groups

Consumer groups can be created and managed through the Azure portal, Azure CLI, Azure PowerShell, or programmatically using the Event Hubs SDKs.

Example: Creating a Consumer Group using Azure CLI


az eventhubs consumer-group create --resource-group <your-resource-group> \
    --namespace-name <your-eventhub-namespace> \
    --event-hub-name <your-eventhub-name> \
    --name <your-new-consumer-group-name>

Example: Accessing Consumer Groups in Code (Conceptual)

When you create an Event Hubs consumer client, you specify the consumer group name:


// Example using .NET SDK
var consumerClient = new EventHubConsumerClient(
    "$Default", // Or your custom consumer group name
    connectionString);

Important Note:

The $Default consumer group is created automatically. While you can use it, it's a best practice to create specific consumer groups for different applications or workloads to ensure better isolation and manageability.

Consumer Group Limits

Azure Event Hubs has limits on the number of consumer groups per Event Hub. Refer to the Azure Event Hubs Quotas and Throttling documentation for the most up-to-date information.

Understanding and effectively utilizing consumer groups is crucial for building robust, scalable, and maintainable event-driven architectures with Azure Event Hubs.