Introduction

Azure Event Hubs is a highly scalable data streaming platform and event ingestion service. This guide provides a comprehensive overview of how to send data to your Event Hubs from various applications and services.

Sending data is a fundamental operation when working with Event Hubs. Whether you are ingesting real-time telemetry, application logs, or clickstream data, understanding the mechanisms for sending events efficiently and reliably is crucial.

Prerequisites

Before you can start sending data, ensure you have the following:

• An Azure subscription.
• An Azure Event Hubs namespace created.
• An Event Hub within the namespace.
• A connection string or appropriate credentials for your Event Hub.

Setting up the SDK

The most common way to interact with Azure Event Hubs is through the official Azure SDKs. Choose the SDK that matches your application's programming language.

Using the .NET SDK

Install the appropriate NuGet package:

dotnet add package Azure.Messaging.EventHubs

Example initialization:

using Azure.Messaging.EventHubs;
using System;
using System.Text;
using System.Threading.Tasks;

// Replace with your actual connection string and hub name
string connectionString = "YOUR_EVENTHUB_CONNECTION_STRING";
string eventHubName = "YOUR_EVENTHUB_NAME";

EventHubProducerClient producer = new EventHubProducerClient(connectionString, eventHubName);

Using the Python SDK

Install the package:

pip install azure-eventhub

Example initialization:

from azure.eventhub import EventHubProducer, EventData

# Replace with your actual connection string and hub name
connection_str = 'YOUR_EVENTHUB_CONNECTION_STRING'
event_hub_name = 'YOUR_EVENTHUB_NAME'

producer = EventHubProducer.from_connection_string(connection_str, event_hub_name)

Sending Individual Events

You can send events one at a time. Each event typically contains a body (payload) and optional properties.

.NET Example

string messageBody = "Hello, Event Hubs!";
EventData eventData = new EventData(Encoding.UTF8.GetBytes(messageBody));
eventData.Properties.Add("Source", "MyApplication");

await producer.SendAsync(new EventData[] { eventData });
Console.WriteLine($"Sent event: {messageBody}");

Python Example

message_body = b"Hello, Event Hubs!"
event_data = EventData(message_body)
event_data.properties[b"Source"] = b"MyApplication"

producer.send(event_data)
print(f"Sent event: {message_body.decode()}")

Sending Batches of Events

For improved efficiency and throughput, it is recommended to send events in batches. This reduces network overhead and improves latency.

.NET Example (using EventDataBatch)

using (EventDataBatch batch = await producer.CreateBatchAsync())
{
    for (int i = 0; i < 5; i++)
    {
        string message = $"Message {i}";
        EventData eventData = new EventData(Encoding.UTF8.GetBytes(message));
        eventData.Properties.Add("BatchIndex", i);

        if (!batch.TryAdd(eventData))
        {
            // The batch is full, send it and start a new one
            Console.WriteLine($"Batch full. Sending {batch.Count} events.");
            await producer.SendAsync(batch);
            batch.Clear(); // Clear to add more events
            // Add the current event to the new batch (if it fits)
            if (!batch.TryAdd(eventData)) {
                Console.WriteLine($"Event {message} is too large to fit in a batch.");
            }
        }
    }

    // Send any remaining events in the last batch
    if (batch.Count > 0)
    {
        Console.WriteLine($"Sending remaining {batch.Count} events.");
        await producer.SendAsync(batch);
    }
}

Python Example (manual batching)

The Python SDK typically handles batching internally when sending multiple EventData objects at once.

events_to_send = []
for i in range(5):
    message = f"Message {i}".encode('utf-8')
    event_data = EventData(message)
    event_data.properties[b"BatchIndex"] = str(i).encode('utf-8')
    events_to_send.append(event_data)

producer.send(events_to_send)
print(f"Sent batch of {len(events_to_send)} events.")

Understanding Partitioning

Event Hubs partitions data across multiple message brokers. When sending events, you can optionally specify a partition key. If a partition key is provided, Event Hubs uses a hash of the key to determine which partition the event should be sent to. This ensures that all events with the same partition key are ordered and arrive in the same partition.

If no partition key is provided, Event Hubs will choose a partition for the event, ensuring load balancing but without guaranteed ordering across events without keys.

Specifying a Partition Key

.NET Example

string partitionKey = "user123"; // e.g., user ID, device ID
string messageBody = "User session data";
EventData eventData = new EventData(Encoding.UTF8.GetBytes(messageBody));
eventData.PartitionKey = partitionKey;

await producer.SendAsync(new EventData[] { eventData });
Console.WriteLine($"Sent event for partition key: {partitionKey}");

Python Example

partition_key = b"user123" # e.g., user ID, device ID
message_body = b"User session data"
event_data = EventData(message_body)
event_data.partition_key = partition_key

producer.send(event_data)
print(f"Sent event for partition key: {partition_key.decode()}")

Error Handling

When sending events, network issues, service throttling, or invalid data can cause operations to fail. Robust error handling is essential.

• Transient Errors: Many errors are transient and can be resolved by retrying the operation. Implement a retry policy with exponential backoff.
• Throttling: If you exceed Event Hubs capacity limits (e.g., throughput units), you may receive throttling errors. Adjust your send rate or scale up your Event Hubs.
• Invalid Data: Ensure your event body is correctly formatted and within size limits.

.NET Example (Basic try-catch)

try
{
    await producer.SendAsync(batch);
    Console.WriteLine("Batch sent successfully.");
}
catch (Exception ex)
{
    Console.WriteLine($"Error sending batch: {ex.Message}");
    // Implement retry logic here
}

Python Example (Basic try-catch)

try:
    producer.send(event_data)
    print("Event sent successfully.")
except Exception as e:
    print(f"Error sending event: {e}")
    # Implement retry logic here

Best Practices

• Batching: Always send events in batches to maximize throughput.
• Partition Keys: Use partition keys to ensure ordering for related events.
• Asynchronous Operations: Use asynchronous methods where available to avoid blocking your application threads.
• Error Handling & Retries: Implement robust error handling and retry mechanisms.
• Connection Management: Reuse the EventHubProducerClient (or equivalent) for efficiency. Avoid creating a new client for every send operation.
• Monitoring: Monitor your Event Hubs metrics (e.g., ingress throughput, egress throughput, throttled requests) in the Azure portal.
• Event Size: Be mindful of the maximum event size (currently 1MB including properties) and batch size limits.