Azure Event Hubs Developer Guide

Core Components

The Event Hubs architecture is built around several key concepts and components:

• Event Hubs Namespace: A logical container for Event Hubs. All Event Hubs exist within a namespace. This provides a unique DNS name for your namespace.
• Event Hub: The actual event stream. An Event Hub is a named entity within a namespace that can ingest and store events.
• Partition: Event Hubs organize data into partitions. Each partition is an ordered, immutable sequence of events. Event Hubs can have multiple partitions (up to 32). Data is written to partitions based on a partition key.
• Consumer Group: A unique view of a populated Event Hub. Each consumer group allows an independent consumption of events from the Event Hub.
• Producer: An application or service that sends events to an Event Hub. Producers can write to specific partitions or let Event Hubs select a partition.
• Consumer: An application or service that reads events from an Event Hub. Consumers belong to consumer groups and read events in order within their assigned partitions.

Data Flow and Ingestion

Event Hubs are designed for high-throughput, low-latency ingestion. The process typically follows these steps:

1. Producers send events: Applications (producers) send event data to a specific Event Hub within a namespace. They can specify a partition key to ensure related events go to the same partition, or Event Hubs can distribute them.
2. Partitioning for Scalability: Events are appended to the end of the specific partition they are routed to. The order of events is guaranteed only within a partition.
3. Durable Storage: Events are stored durably in Event Hubs for a configurable retention period.
4. Consumers read events: Applications (consumers) belonging to consumer groups connect to the Event Hub and read events. They track their position (offset) within each partition they are consuming from.

This partitioning strategy allows Event Hubs to scale horizontally, handling millions of events per second.

Key Architectural Concepts

Scalability and Throughput

Event Hubs achieve high throughput through partitioning. Each partition can be independently scaled. By increasing the number of partitions, you increase the overall throughput of the Event Hub.

Event Hubs offer different tiers (Basic, Standard, Premium) that provide varying levels of throughput units (TUs) and ingress/egress traffic.

Durability and Retention

Events are stored durably. The retention period can be configured from 24 hours up to 7 days for Basic and Standard tiers, and up to 90 days for Premium tiers (with some restrictions). This allows consumers to process events even if they are temporarily offline.

Ordered Delivery

Within a single partition, events are guaranteed to be delivered in the order they were received. This is critical for many scenarios, such as maintaining state consistency.

At-Least-Once Delivery

Event Hubs guarantees at-least-once delivery. This means that an event will be delivered at least once, but it's possible for duplicate events to be delivered. Consumers must be designed to handle potential duplicates idempotently.

Consumer Groups

Consumer groups provide a flexible way to have multiple applications or services read from the same Event Hub independently without interfering with each other. Each consumer group maintains its own offset for each partition.

Integration with Azure Services

Event Hubs integrate seamlessly with other Azure services:

• Azure Stream Analytics: For real-time data processing and analytics.
• Azure Functions: For serverless event-driven compute.
• Azure Databricks: For large-scale data processing and machine learning.
• Azure Data Lake Storage: For archiving massive amounts of data.
• Azure Cosmos DB: For NoSQL data management.

Code Example: Sending an Event (Conceptual)

Here's a conceptual example of how a producer might send an event using the Azure SDK for .NET:

using Azure.Messaging.EventHubs;
using Azure.Messaging.EventHubs.Producer;
using System;
using System.Text;
using System.Threading.Tasks;

// Replace with your Event Hubs namespace and event hub name
string eventHubNamespace = "your-event-hubs-namespace.servicebus.windows.net";
string eventHubName = "your-event-hub-name";
string connectionString = "Endpoint=sb://your-event-hubs-namespace.servicebus.windows.net/..."; // Your connection string

await using var producerClient = new EventHubProducerClient(connectionString, eventHubName);

string messageBody = "{\"temperature\": 25.5, \"humidity\": 60}";
var eventData = new EventData(Encoding.UTF8.GetBytes(messageBody));

try
{
    await producerClient.SendEventAsync(eventData);
    Console.WriteLine($"Sent event: {messageBody}");
}
catch (Exception ex)
{
    Console.WriteLine($"Error sending event: {ex.Message}");
}