Managing Azure Event Hubs

Overview

Effective management of your Azure Event Hubs is crucial for maintaining a robust and scalable event streaming platform. This section covers common management tasks, including creation, configuration, and deletion of Event Hubs and their associated resources.

You can manage Event Hubs using various tools:

Azure Portal: A visual interface for managing resources.
Azure CLI: A command-line tool for automating management tasks.
Azure SDKs: Programmatic access for integrating management into applications.
ARM Templates/Bicep: For infrastructure as code deployments.

Best Practice: For production environments, consider using Infrastructure as Code (IaC) tools like ARM Templates or Bicep to ensure consistent and repeatable deployments of your Event Hubs namespaces and Event Hubs.

Creating Event Hubs

Before you can create an Event Hub, you need an Event Hubs namespace. A namespace is a container for Event Hubs.

Creating an Event Hubs Namespace

You can create a namespace using the Azure Portal or Azure CLI.

Using Azure Portal

Navigate to the Azure Portal and search for "Event Hubs".
Click "Create".
Select your subscription and resource group.
Provide a unique namespace name.
Choose a region.
Select a pricing tier (Basic, Standard, Premium). Standard tier is recommended for most production workloads.
Review and create.

Using Azure CLI

Use the following command:


az eventhubs namespace create --resource-group <YourResourceGroup> --name <YourNamespaceName> --location <YourRegion> --sku Standard

Creating an Event Hub within a Namespace

Once a namespace is created, you can create one or more Event Hubs inside it.

Using Azure Portal

Navigate to your Event Hubs namespace in the Azure Portal.
Under "Entities", select "Event Hubs".
Click "+ Event Hub".
Enter the name for your Event Hub.
Configure settings like the number of partitions and message retention period.
Click "Create".

Using Azure CLI

Use the following command:


az eventhubs create --resource-group <YourResourceGroup> --namespace-name <YourNamespaceName> --name <YourEventHubName> --partitions 4 --message-retention 7

--partitions: The number of partitions. This is crucial for parallelism in event processing.

--message-retention: The duration in days for which event data is retained.

Note: The maximum number of partitions for a Standard tier namespace is 32, and for Premium tier it's 1024. Basic tier supports a maximum of 2 partitions.

Configuring Event Hubs

Event Hubs offer several configuration options to tailor their behavior to your needs.

Configuring Partitions

Partitions are a fundamental aspect of Event Hubs, influencing throughput and parallelism. You set the number of partitions during creation and cannot change it afterward. If you need to increase partitions, you must create a new Event Hub with the desired partition count and migrate data.

Configuring Message Retention

Message retention determines how long events are stored in an Event Hub. This is configurable and can be updated.

Azure Portal: Navigate to your Event Hub, then "Settings" -> "General". You can adjust the "Message retention" slider.
Azure CLI:


az eventhubs update --resource-group <YourResourceGroup> --namespace-name <YourNamespaceName> --name <YourEventHubName> --message-retention 14

Configuring Throughput (Standard & Premium Tiers)

For Standard and Premium tiers, you can configure throughput units (TUs) or processing units (PUs) to control ingress and egress capacity.

Standard Tier: Configured via Throughput Units (TUs).
Premium Tier: Configured via Processing Units (PUs).

Adjusting these settings can be done via the Azure Portal under the Event Hubs namespace settings or using the Azure CLI. Scaling up TUs/PUs generally incurs higher costs but provides more capacity.

Configuring Capture

Azure Event Hubs Capture allows you to automatically and incrementally batch events from an Event Hub and write them to Azure Blob Storage or Azure Data Lake Storage. This is useful for archival and downstream batch processing.

Azure Portal: Navigate to your Event Hub, then "Settings" -> "Event Hubs Capture". Enable capture and configure the destination storage account and container.
Azure CLI:


az eventhubs capture create --resource-group <YourResourceGroup> --namespace-name <YourNamespaceName> --event-hub-name <YourEventHubName> --enabled true --destination-storage-account <YourStorageAccountName> --destination-container <YourContainerName>

Configuring Geo-Disaster Recovery

For high availability, you can configure Geo-disaster recovery. This involves setting up a primary and secondary Event Hubs namespace in different regions. Writes to the primary namespace are automatically replicated to the secondary.

This is configured at the namespace level in the Azure Portal under "Disaster Recovery".

Deleting Event Hubs and Namespaces

Deleting resources is a critical management operation. Ensure you have backed up any necessary data before proceeding.

Deleting an Event Hub

You can delete an individual Event Hub from within its namespace.

Azure Portal: Navigate to your Event Hubs namespace, select "Event Hubs", click on the Event Hub you wish to delete, and then click the "Delete" button.
Azure CLI:


az eventhubs delete --resource-group <YourResourceGroup> --namespace-name <YourNamespaceName> --name <YourEventHubName>

Deleting an Event Hubs Namespace

Deleting a namespace will also delete all Event Hubs and consumer groups within it.

Azure Portal: Navigate to your Event Hubs namespace and click the "Delete" button at the top.
Azure CLI:


az eventhubs namespace delete --resource-group <YourResourceGroup> --name <YourNamespaceName>

Warning: Deletion operations are irreversible. Once a resource is deleted, it cannot be recovered. Always double-check the resource you are deleting and ensure you have a backup strategy in place if the data is critical.