Streaming Data with Azure Event Hubs

Introduction

Azure Event Hubs is a highly scalable data streaming platform and event ingestion service. It can capture, transform, and store millions of events per second. This tutorial will walk you through the fundamental steps of setting up and using Event Hubs to stream data.

We'll cover:

Creating an Event Hubs namespace and an event hub.
Sending sample events to the event hub.
Reading events from the event hub using a consumer group.

Prerequisites

An active Azure subscription. If you don't have one, you can create a free account.
Azure CLI installed and configured, or use Azure Cloud Shell.
.NET Core SDK or Node.js installed (for sample code).

Step 1: Create an Event Hubs Namespace and Event Hub

First, we need a namespace, which is a container for all our Event Hubs instances. Then, we create an event hub within that namespace.

Using Azure CLI

Replace YourResourceGroup and YourNamespaceName with your desired names.

az group create --name YourResourceGroup --location eastus
az eventhubs namespace create --resource-group YourResourceGroup --name YourNamespaceName --location eastus --sku Standard
az eventhubs eventhub create --resource-group YourResourceGroup --namespace-name YourNamespaceName --name myeventhub
            

Using Azure Portal

Navigate to the Azure portal.
Search for "Event Hubs" and select it.
Click "Create" to start the namespace creation.
Fill in the required details (Subscription, Resource group, Region, Name, Pricing tier).
Once the namespace is created, navigate to it.
Click on "+ Event Hub" to create an event hub within your namespace.

Step 2: Get Connection Strings

To send or receive data, your applications will need a connection string. You can get it from the portal or via CLI.

Using Azure CLI

az eventhubs namespace authorization-rule list --resource-group YourResourceGroup --namespace-name YourNamespaceName
            

Look for the primary key of the RootManageSharedAccessKey or create a specific policy with send/listen rights.

Using Azure Portal

Navigate to your Event Hubs namespace.
Under "Settings", click on "Shared access policies".
Select a policy (e.g., RootManageSharedAccessKey) and copy the "Primary Connection String".

Step 3: Send Events

Let's send some sample data to our event hub. We'll use a simple C# example.

C# Example

Install the Azure.Messaging.EventHubs NuGet package.

dotnet add package Azure.Messaging.EventHubs

Create a file named Sender.cs:

using Azure.Messaging.EventHubs;
using Azure.Messaging.EventHubs.Producer;
using System;
using System.Text;
using System.Threading.Tasks;

namespace EventHubsSender
{
    class Program
    {
        private const string eventHubsConnectionString = "YOUR_EVENT_HUBS_CONNECTION_STRING";
        private const string eventHubName = "myeventhub";

        static async Task Main(string[] args)
        {
            await using var producerClient = new EventHubProducerClient(eventHubsConnectionString, eventHubName);

            using EventDataBatch eventDataBatch = await producerClient.CreateBatchAsync();

            for (int i = 1; i <= 5; i++)
            {
                var eventBody = $"Message {i}";
                if (!eventDataBatch.TryAddMessage(new EventData(Encoding.UTF8.GetBytes(eventBody))))
                {
                    throw new Exception($"The message {eventBody} is too large for the batch.");
                }
            }

            try
            {
                await producerClient.SendAsync(eventDataBatch);
                Console.WriteLine("A batch of 5 events has been sent.");
            }
            catch (Exception ex)
            {
                Console.WriteLine($"Error sending batch: {ex.Message}");
            }

            Console.WriteLine("Press any key to exit.");
            Console.ReadKey();
        }
    }
}
            

Replace YOUR_EVENT_HUBS_CONNECTION_STRING with the connection string obtained in Step 2. Then, run the application:

dotnet run

Step 4: Receive Events

Now, let's create a simple consumer to read the events we just sent.

C# Example

Create a file named Receiver.cs:

using Azure.Messaging.EventHubs;
using Azure.Messaging.EventHubs.Consumer;
using System;
using System.Text;
using System.Threading.Tasks;
using System.Collections.Generic;

namespace EventHubsReceiver
{
    class Program
    {
        private const string eventHubsConnectionString = "YOUR_EVENT_HUBS_CONNECTION_STRING";
        private const string eventHubName = "myeventhub";
        private const string consumerGroup = EventHubConsumerClient.DefaultConsumerGroupName; // Or a custom group

        static async Task Main(string[] args)
        {
            await using var consumerClient = new EventHubConsumerClient(consumerGroup, eventHubsConnectionString, eventHubName);

            Console.WriteLine("Listening for events...");

            await foreach (PartitionEventPartition currentPartition in consumerClient.ReadEventsAsync())
            {
                Console.WriteLine($"Processing event from partition: {currentPartition.PartitionId}");
                foreach (EventData eventData in currentPartition.Events)
                {
                    string messageBody = Encoding.UTF8.GetString(eventData.EventBody.ToArray());
                    Console.WriteLine($"  Message: {messageBody}");
                }
            }
        }
    }
}
            

Replace YOUR_EVENT_HUBS_CONNECTION_STRING and run the receiver application:

dotnet run

You should see the messages you sent being printed to the console.

Next Steps

This tutorial covered the basics. For more advanced scenarios, consider exploring:

Event Hubs Partitioning for parallel processing.
Integrating with Azure Functions for serverless event processing.
Using Azure Stream Analytics for real-time analytics.
Implementing robust error handling and retry mechanisms.
Managing consumer groups for different applications reading from the same hub.