Azure Event Hubs Developer's Guide

Sending Data to Azure Event Hubs

This guide covers the essential steps and best practices for sending data to Azure Event Hubs. Event Hubs is designed for high-throughput telemetry ingestion, making it a cornerstone for many big data and real-time analytics solutions.

Choosing Your Sending Method

Azure Event Hubs supports sending data through various methods:

• Azure SDKs: Recommended for most scenarios. Available for popular languages like C#, Java, Python, JavaScript, and Go. They abstract away complexities and provide a robust API.
• Apache Kafka Protocol: If you are migrating from Apache Kafka or have existing Kafka clients, Event Hubs for Kafka provides a Kafka endpoint.
• REST API: For simple integrations or environments where SDKs are not feasible.

Using the Azure SDK (Python Example)

The Azure SDKs simplify the process of connecting to and sending data to Event Hubs. Here's a basic example using the Python SDK:

from azure.eventhub import EventHubProducerClient, EventData

# Replace with your actual connection string and event hub name
CONNECTION_STR = "YOUR_EVENT_HUB_CONNECTION_STRING"
EVENT_HUB_NAME = "YOUR_EVENT_HUB_NAME"

def send_events():
    # Create a producer client
    producer = EventHubProducerClient.from_connection_string(
        CONNECTION_STR, EVENT_HUB_NAME
    )

    try:
        # Create a batch of events
        event_batch = producer.create_batch()

        # Add events to the batch
        for i in range(5):
            message = f"Event {i}: This is a sample message."
            event_batch.add(EventData(message))

        # Send the batch of events
        producer.send_batch(event_batch)
        print("Successfully sent a batch of events.")

        # Send individual events if not batching
        # producer.send_event(EventData("Another single event"))
        # print("Sent a single event.")

    except Exception as e:
        print(f"An error occurred: {e}")
    finally:
        # Close the producer client
        producer.close()

if __name__ == "__main__":
    send_events()
                

For detailed examples in other languages, please refer to the SDK References.

Event Structure

Events sent to Event Hubs are typically JSON or Avro payloads. While Event Hubs itself doesn't enforce a schema, your downstream consumers will likely expect a consistent structure. Common practice includes adding metadata such as timestamps, source identifiers, and event types.

event_batch.add(EventData(message), partition_key="user_session_123")
                    

Batching for Efficiency

Sending events one by one can be inefficient due to network overhead. Batching allows you to group multiple events together and send them in a single request. The Azure SDKs provide methods to create and manage event batches.

• Benefits of Batching: Reduced latency, lower cost, improved throughput.
• Batch Size Limits: Event Hubs has limits on the maximum size of an event batch (e.g., 1MB). The SDKs can help manage this.

Sending to Specific Partitions

You can directly send events to a specific partition by providing the partition ID when creating the batch or sending individual events, though using partition keys is generally preferred for managing data distribution.

Sending via REST API

If you need to send data using the REST API, you'll typically use a POST request to the Event Hubs endpoint with an authorization header. The request body will contain your events, often Base64 encoded.

POST /your-event-hub-namespace.servicebus.windows.net/your-event-hub-name/messages?api-version=2014-01
Content-Type: application/vnd.microsoft.eventhub.management.v2014-01+json
Authorization: SharedAccessSignature sr=...&sig=...&se=...&skn=...

[
  {
    "body": "eyJlbmV.g.C.V.Z.T.z.Iu.IjogIjIwMjMtMTIuLjAuMTYuMTIuMzkuMTIyMTA2MloiLCAi.aW5mbyI6ICJIZWxsbywgRXZlbnQgSHViMSEifQ==",
    "properties": {
      "correlationId": "12345"
    }
  }
]
                

Note: Managing authorization signatures for the REST API can be complex. SDKs are highly recommended.

Next Steps

Now that you know how to send data, explore how to efficiently receive and process it. Learn about Receiving Data.