Azure Event Hubs

Microsoft's highly scalable data streaming platform and event ingestion service.

Introduction to Azure Event Hubs for Kafka Users

Azure Event Hubs provides a highly scalable, fully managed, real-time data streaming platform. For developers familiar with Apache Kafka, Event Hubs offers a compatible endpoint, allowing you to leverage your existing Kafka applications and expertise with the benefits of a cloud-native, managed service.

This document is designed to guide Kafka users through understanding, migrating to, and utilizing Azure Event Hubs. We'll cover the compatibility aspects, key benefits, and practical steps to get you started.

Benefits of Using Event Hubs for Kafka

Kafka Compatibility

Event Hubs implements the Kafka endpoint protocol, allowing existing Kafka clients to connect to Event Hubs with minimal to no changes. This means you can use your preferred Kafka libraries and tools with Event Hubs.

The primary mechanism for compatibility is the Kafka endpoint provided by your Event Hubs namespace. When you enable the Kafka endpoint, Event Hubs exposes a bootstrap server address that your Kafka clients can connect to.

Key Kafka concepts map to Event Hubs as follows:

Note: While Event Hubs is highly compatible, there might be subtle differences in behavior or feature support compared to a self-managed Kafka cluster. Always refer to the official Azure documentation for the most up-to-date information on compatibility details and limitations.

Getting Started

  1. Create an Azure Account: If you don't have one, sign up for a free Azure account.
  2. Create an Event Hubs Namespace: In the Azure portal, create a new Event Hubs namespace.
  3. Enable Kafka Endpoint: Within your Event Hubs namespace settings, find and enable the Kafka endpoint. This will provide you with the necessary bootstrap server address and authentication credentials.
  4. Configure Kafka Clients: Update your Kafka producer and consumer configurations to point to the Event Hubs bootstrap server and use the provided credentials.
  5. Create an Event Hub: Within your namespace, create an Event Hub (which corresponds to a Kafka topic).

Usage Patterns

Migrating Kafka Applications

For existing Kafka applications, the migration process is often straightforward:

  1. Update Bootstrap Servers: Change the bootstrap.servers property in your Kafka producer/consumer configuration to the Event Hubs Kafka endpoint address.
  2. Update Security Settings: Configure your clients to use SAS (Shared Access Signatures) or Azure Active Directory for authentication. SAS keys are often the easiest to start with.
  3. Test Thoroughly: Run your applications against Event Hubs to ensure correct behavior and performance.

Building New Applications

When building new applications designed for the cloud:

  1. Leverage Kafka SDKs: Continue using familiar Kafka client libraries.
  2. Design for Scalability: Utilize Event Hubs' partitioning and consumer groups to distribute load effectively.
  3. Integrate with Azure Services: Explore seamless integration with other Azure services for advanced processing and analytics.

Configuration Examples

Here are examples of how you might configure your Kafka clients to connect to Azure Event Hubs.

Kafka Producer Configuration (Java Example)


import org.apache.kafka.clients.producer.*;
import java.util.Properties;

public class EventHubsProducer {
    public static void main(String[] args) throws Exception {
        Properties props = new Properties();
        props.put("bootstrap.servers", "YOUR_EVENTHUBS_NAMESPACE.servicebus.windows.net:9093");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

        // SAS authentication (replace with your actual SAS key name and value)
        // Format: "SharedAccessKeyName=YOUR_SAS_KEY_NAME;SharedAccessKey=YOUR_SAS_KEY_VALUE"
        props.put("security.protocol", "SASL_SSL");
        props.put("sasl.mechanism", "PLAIN");
        props.put("sasl.jaas.config", "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"$ConnectionString\" password=\"Endpoint=sb://YOUR_EVENTHUBS_NAMESPACE.servicebus.windows.net/;SharedAccessKeyName=YOUR_SAS_KEY_NAME;SharedAccessKey=YOUR_SAS_KEY_VALUE\";");

        Producer<String, String> producer = new KafkaProducer<>(props);

        for (int i = 0; i < 100; i++) {
            String message = "Message number " + i;
            producer.send(new ProducerRecord<>("your-eventhub-name", Integer.toString(i), message));
            System.out.println("Sent: " + message);
        }

        producer.close();
    }
}

Kafka Consumer Configuration (Java Example)


import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;

public class EventHubsConsumer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "YOUR_EVENTHUBS_NAMESPACE.servicebus.windows.net:9093");
        props.put("group.id", "my-consumer-group"); // Your consumer group ID
        props.put("key.deserializer", StringDeserializer.class.getName());
        props.put("value.deserializer", StringDeserializer.class.getName());
        props.put("auto.offset.reset", "earliest"); // Start from the beginning of the log

        // SAS authentication
        props.put("security.protocol", "SASL_SSL");
        props.put("sasl.mechanism", "PLAIN");
        props.put("sasl.jaas.config", "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"$ConnectionString\" password=\"Endpoint=sb://YOUR_EVENTHUBS_NAMESPACE.servicebus.windows.net/;SharedAccessKeyName=YOUR_SAS_KEY_NAME;SharedAccessKey=YOUR_SAS_KEY_VALUE\";");

        Consumer<String, String> consumer = new KafkaConsumer<>(props);
        consumer.subscribe(Collections.singletonList("your-eventhub-name")); // The Event Hub name

        System.out.println("Starting consumer...");

        while (true) {
            ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
            for (ConsumerRecord<String, String> record : records) {
                System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
            }
            // Commit offsets periodically if auto.commit.enable is false
            // consumer.commitSync();
        }
        // consumer.close(); // This loop is infinite in this example
    }
}
Security Note: Never embed SAS keys directly in client-side code for production applications. Use secure methods for credential management like Azure Key Vault, environment variables, or managed identities.

Key Considerations

Next Steps