Azure Event Hubs

Events

An event is a small unit of information. In the context of Event Hubs, an event is typically a record of something that has happened. Events are immutable and are organized into ordered sequences.

An event consists of:

• Body: The payload of the event, which can be any type of data (e.g., JSON, Avro, plain text).
• Properties: Key-value pairs that provide metadata about the event.
• System Properties: Properties assigned by Event Hubs, such as the partition key and offset.

Producers

Producers are applications or services that send (publish) events to an Event Hub. They are responsible for generating and sending data to the event stream.

Producers can:

• Send events individually or in batches.
• Specify a partition key to influence which partition an event is sent to.
• Use different SDKs (e.g., .NET, Java, Python, JavaScript) to interact with Event Hubs.

Example of sending an event (conceptual):

// Pseudocode
            eventHubClient.send({
                body: JSON.stringify({ sensorId: "temp-001", value: 25.5 }),
                partitionKey: "sensor-group-abc"
            });

Consumers

Consumers are applications or services that read (subscribe to) events from an Event Hub. They process the stream of events for various purposes, such as analytics, data warehousing, or triggering actions.

Consumers use consumer groups to read events. Each consumer group maintains its own state and reads events independently from other consumer groups.

Consumer Groups

A consumer group is a view of an Event Hub. Each consumer group allows multiple applications to read from an Event Hub independently without blocking each other.

Key characteristics:

• When an application joins a consumer group, it starts reading events from the earliest available offset or a specified offset.
• Event Hubs retains event data for a configurable period (default 1 day, up to 7 days). Consumers must read within this retention period.
• If no consumer group exists for an application, a default one ($Default) is created automatically.

Consider creating separate consumer groups for different processing applications (e.g., one for real-time dashboard, another for batch analytics).

Partitions

An Event Hub is divided into one or more partitions. Partitions are ordered, immutable sequences of events.

When events are sent to an Event Hub, they are distributed across these partitions. The partitioning strategy ensures:

• Scalability: Different partitions can be processed in parallel by different consumer instances.
• Ordering: Events within a single partition are always in the order they were received.

Producers can influence partitioning by providing a partition key. If a partition key is provided, all events with the same key will land in the same partition, ensuring order for events related to the same entity (e.g., a specific sensor or user).

If no partition key is provided, Event Hubs assigns an event to a partition, typically in a round-robin fashion.

Offsets

An offset is a unique, continuous number that Event Hubs assigns to each event as it arrives. The offset serves as a pointer to the position of an event within a partition.

Consumers use offsets to track their progress. When a consumer reads events, it commits the offset of the last successfully processed event. If a consumer restarts, it can resume reading from the last committed offset, ensuring no events are lost or processed twice.