Azure Event Hubs Documentation

Event Hubs Messaging Concepts

Introduction to Event Hubs Messaging

Azure Event Hubs is a highly scalable data streaming platform and event ingestion service. It can capture millions of events per second so you can develop more applications and protect your data. Event Hubs is built on the Apache Kafka protocol. It enables you to process streams of events in real-time.

Core Messaging Components

Event Hubs revolves around a few key components that define how data is ingested, processed, and consumed:

• Event Producers: Applications or services that send event data to an Event Hub.
• Event Consumers: Applications or services that read event data from an Event Hub.
• Event Hub: A managed service that acts as the central message broker for event streams.
• Partition: A logical stream within an Event Hub. Events are appended to partitions in order.

Event Structure

An event is a unit of information published to Event Hubs. It typically consists of:

• Body: The actual data payload of the event. This can be any format (JSON, Avro, binary, etc.).
• Properties: Metadata associated with the event, such as content type, correlation IDs, or custom application-specific properties.
• System Properties: Properties set by Event Hubs, such as offset, sequence number, and partition key.

{
  "body": {
    "deviceId": "sensor-001",
    "timestamp": "2023-10-27T10:00:00Z",
    "temperature": 25.5,
    "humidity": 60
  },
  "properties": {
    "contentType": "application/json",
    "operationId": "abc123xyz789"
  },
  "systemProperties": {
    "offset": 123456,
    "sequenceNumber": 9876,
    "partitionKey": "sensor-001"
  }
}
                

Publishing and Consuming Events

Producers send events to an Event Hub. Consumers then read these events. Event Hubs ensures that events are delivered reliably and in order within each partition.

• Ordered Delivery: Events within a single partition are always delivered to consumers in the order they were received by the producer.
• At-Least-Once Delivery: Event Hubs guarantees that each event will be delivered at least once. Consumers must be designed to handle potential duplicate events (e.g., using idempotency).

Partitioning Strategy

Partitions are crucial for scalability and parallel processing. When a producer sends an event, it can specify a partition key. If a partition key is provided, Event Hubs ensures that all events with the same partition key are sent to the same partition. If no partition key is specified, Event Hubs distributes events across partitions in a round-robin fashion.

This partitioning strategy allows consumers to process events in parallel. For example, if you have 10 partitions and 5 consumer instances, you can assign partitions to consumers for highly parallel processing.

Consumer Groups

A consumer group is a named view of a consumer for an Event Hub. Each consumer group maintains its own position in the event stream and can read events independently of other consumer groups. This allows multiple applications or services to consume the same event stream without interfering with each other.

By default, an Event Hub has a default consumer group. You can create additional consumer groups to support various processing needs.

Key Concepts Summary

• Events: The fundamental unit of data.
• Producers: Send events.
• Consumers: Read events.
• Partitions: Ordered streams for scalability and parallelism.
• Partition Key: Guarantees event affinity to a specific partition.
• Consumer Groups: Independent views of an event stream for multiple applications.