Sending Events to Azure Event Hubs
This guide details how to send events to Azure Event Hubs using various programming languages and SDKs. Event Hubs is a highly scalable data streaming platform and event ingestion service. Producers send data to Event Hubs, which then stores the data and makes it available for consumption.
Choosing Your SDK
Azure Event Hubs provides robust SDKs for multiple languages, enabling seamless integration into your applications. The most common choices include:
Core Concepts for Sending
- Event Hubs Namespace: A container for all your Event Hubs instances.
- Event Hub: The actual data stream where events are sent.
- Partition: Event Hubs are divided into partitions, which allow for parallel processing and scale. Events sent to a specific partition are ordered within that partition.
- Partition Key: A string value used by Event Hubs to determine which partition an event should be sent to. If a partition key is provided, all events with the same partition key will be sent to the same partition, ensuring order. If not provided, Event Hubs will use round-robin distribution.
- Connection String: Used to authenticate with your Event Hubs namespace.
Important: Always secure your connection strings and avoid embedding them directly in client-side code. Use Azure Key Vault or environment variables for secure management.
Sending Events with .NET SDK
The Azure SDK for .NET provides the Azure.Messaging.EventHubs package for interacting with Event Hubs.
Prerequisites
- Install the NuGet package:
Install-Package Azure.Messaging.EventHubs - Obtain your Event Hubs connection string and the Event Hub name.
Example: Sending a Single Event
using Azure.Messaging.EventHubs;
using System;
using System.Text;
using System.Threading.Tasks;
// Your connection string and event hub name
string connectionString = "YOUR_EVENT_HUBS_CONNECTION_STRING";
string eventHubName = "YOUR_EVENT_HUB_NAME";
await using var producerClient = new EventHubProducerClient(connectionString, eventHubName);
// Create an event batch
using EventDataBatch eventBatch = await producerClient.CreateBatchAsync();
var eventData = new EventData(Encoding.UTF8.GetBytes("Hello, Event Hubs!"));
if (!eventBatch.TryAddMessage(eventData))
{
// The batch is full, send it and create a new one
await producerClient.SendAsync(eventBatch);
Console.WriteLine("Batch sent.");
// Create a new batch for the next message
eventBatch = await producerClient.CreateBatchAsync();
eventData = new EventData(Encoding.UTF8.GetBytes("Another event!"));
if (!eventBatch.TryAddMessage(eventData))
{
throw new Exception("Event too large for a single batch.");
}
}
// Send the remaining events in the batch
await producerClient.SendAsync(eventBatch);
Console.WriteLine("Final batch sent.");
Console.WriteLine("Events sent successfully!");
Example: Sending Events with Partition Key
using Azure.Messaging.EventHubs;
using System;
using System.Text;
using System.Threading.Tasks;
string connectionString = "YOUR_EVENT_HUBS_CONNECTION_STRING";
string eventHubName = "YOUR_EVENT_HUB_NAME";
await using var producerClient = new EventHubProducerClient(connectionString, eventHubName);
using EventDataBatch eventBatch = await producerClient.CreateBatchAsync();
// Example with partition key
var eventDataWithKey = new EventData(Encoding.UTF8.GetBytes("Event for User123"));
eventDataWithKey.PartitionKey = "User123"; // Ensures this event goes to the same partition as other "User123" events
if (!eventBatch.TryAddMessage(eventDataWithKey))
{
await producerClient.SendAsync(eventBatch);
// Handle error if needed
}
await producerClient.SendAsync(eventBatch);
Console.WriteLine("Events with partition key sent!");
Sending Events with Java SDK
The Azure SDK for Java provides the azure-messaging-eventhubs library.
Prerequisites
- Add the dependency to your
pom.xml(for Maven):<dependency> <groupId>com.azure</groupId> <artifactId>azure-messaging-eventhubs</artifactId> <version>5.13.0</version><!-- Check for the latest version --> </dependency> - Obtain your Event Hubs connection string and the Event Hub name.
Example: Sending a Batch of Events
import com.azure.messaging.eventhubs.EventData;
import com.azure.messaging.eventhubs.EventHubClientBuilder;
import com.azure.messaging.eventhubs.EventSender;
import com.azure.messaging.eventhubs.models.CreateBatchOptions;
import java.util.ArrayList;
import java.util.List;
public class EventHubsSender {
public static void main(String[] args) {
String connectionString = "YOUR_EVENT_HUBS_CONNECTION_STRING";
String eventHubName = "YOUR_EVENT_HUB_NAME";
// Create an EventSender client using a connection string
EventSender eventSender = new EventHubClientBuilder()
.connectionString(connectionString, eventHubName)
.buildEventSender();
// Create a batch of events
CreateBatchOptions options = new CreateBatchOptions()
.setMaxSizeInBytes(1 * 1024 * 1024); // 1MB max size
// Note: Creating the batch inside a loop is necessary if you need to ensure
// that individual events are not too large to fit in the batch.
// For simplicity, we are creating one batch here.
List<EventData> eventsToSend = new ArrayList<>();
eventsToSend.add(new EventData("Hello Event Hubs from Java!".getBytes()));
eventsToSend.add(new EventData("Another message.".getBytes()));
try {
// Try to add events to the batch. The send() method can take a list of EventData directly.
// If you need finer control over batching, you would use eventSender.createBatch()
// and add messages one by one using eventBatch.tryAdd().
// For simplicity and common use cases, we use the simpler send() overload.
eventSender.send(eventsToSend).block(); // block() makes it synchronous
System.out.println("Events sent successfully!");
} catch (Exception e) {
System.err.println("Error sending events: " + e.getMessage());
e.printStackTrace();
} finally {
// Close the sender client
eventSender.close();
}
}
}
Sending Events with Python SDK
The Azure SDK for Python includes the azure-eventhubs library.
Prerequisites
- Install the library:
pip install azure-eventhubs - Obtain your Event Hubs connection string and the Event Hub name.
Example: Sending Events Asynchronously
import asyncio
from azure.eventhub import EventData, EventHubProducerClient
async def send_events():
connection_str = "YOUR_EVENT_HUBS_CONNECTION_STRING"
eventhub_name = "YOUR_EVENT_HUB_NAME"
# Create a producer client using the connection string
producer = EventHubProducerClient.from_connection_string(
connection_str, eventhub_name
)
async with producer:
# Create a batch of events
batch = await producer.create_batch()
batch.add_event(EventData("Hello Event Hubs from Python!"))
batch.add_event(EventData("Another event here."))
# Send the batch
await producer.send_batch(batch)
print("Events sent successfully!")
# Example with partition key
batch_with_key = await producer.create_batch()
event_with_key = EventData("Event for Partition A")
event_with_key.partition_key = "PartitionA"
batch_with_key.add_event(event_with_key)
await producer.send_batch(batch_with_key)
print("Event with partition key sent!")
if __name__ == "__main__":
asyncio.run(send_events())
Sending Events with Node.js SDK
The Azure SDK for JavaScript provides the @azure/event-hubs package.
Prerequisites
- Install the package:
npm install @azure/event-hubs - Obtain your Event Hubs connection string and the Event Hub name.
Example: Sending Events Asynchronously
const { EventHubProducerClient, EventData } = require("@azure/event-hubs");
async function sendEvents() {
const connectionString = "YOUR_EVENT_HUBS_CONNECTION_STRING";
const eventHubName = "YOUR_EVENT_HUB_NAME";
const producerClient = new EventHubProducerClient(connectionString, eventHubName);
// Create a batch of events
const producer = producerClient.createProducer();
const batch = await producer.createBatch();
batch.add({ body: "Hello Event Hubs from Node.js!" });
batch.add({ body: "Yet another message." });
// Send the batch
await producer.send(batch);
console.log("Events sent successfully!");
// Example with partition key
const batchWithKey = await producer.createBatch();
batchWithKey.add({ body: "Event for Partition B", partitionKey: "PartitionB" });
await producer.send(batchWithKey);
console.log("Event with partition key sent!");
await producer.close();
await producerClient.close();
}
sendEvents().catch((err) => {
console.error("Error sending events:", err);
});
Batching Events
For efficiency, it's recommended to batch multiple events together before sending them to Event Hubs. The SDKs provide methods to create batches and add events. Each batch has a size limit (e.g., 1MB). If an event exceeds the maximum batch size, you'll need to handle it (e.g., by splitting it or logging an error).
Tip: Batching reduces the number of network requests, leading to better throughput and lower costs.
Sending to Specific Partitions
To ensure events are processed in order within a specific partition, you can specify a PartitionKey when creating or sending an event. All events with the same partition key will be routed to the same partition.
- If you need strong ordering for a specific entity (e.g., all events for a particular user), use a partition key derived from that entity's ID.
- If you don't specify a partition key, Event Hubs will distribute events across partitions using a round-robin algorithm.