Azure Documentation

Understanding Offsets in Azure Event Hubs

Offsets are fundamental to how Event Hubs consumers track their progress within an event stream. This document explains what offsets are, how they work, and best practices for managing them.

What are Offsets?

In Azure Event Hubs, each event in a partition is assigned a unique, sequential, 64-bit integer value called an offset. Offsets are strictly ordered within a partition and serve as a logical position marker for events.

When an application reads events from an Event Hub partition, it keeps track of the offset of the last event it successfully processed. This offset is then used to resume reading from that exact point if the consumer application restarts or encounters an error.

Offset Characteristics:

  • Unique: Each offset is unique within a partition.
  • Sequential: Offsets increase incrementally for each new event.
  • Immutable: Once assigned, an offset does not change.
  • Partition-Scoped: Offsets are only meaningful within the context of a specific partition.
  • Client-Managed: Consumers are responsible for storing and managing offsets.

How Offsets Work with Consumers

Event Hubs consumers typically operate in one of two modes:

  1. At-Least-Once Processing: The consumer commits the offset after successfully processing an event. If the consumer crashes before committing, it might re-process the same event upon restart.
  2. At-Most-Once Processing: The consumer commits the offset before processing the event. This prevents reprocessing but risks losing events if the consumer crashes between committing the offset and processing the event.

The choice between these modes depends on your application's tolerance for duplicates versus data loss. For most scenarios, at-least-once processing with proper de-duplication logic in the application is preferred.

Consumer Groups and Offsets

Each consumer group in an Event Hub maintains its own set of offsets for each partition. This isolation ensures that different consumer groups can independently read and process the event stream without interfering with each other.

Managing Offsets

The responsibility of storing and retrieving offsets lies with the consumer application. Common strategies include:

  • Azure Storage: Storing offsets in Azure Blob Storage or Azure Table Storage is a robust and scalable option.
  • Azure Cosmos DB: For applications already using Cosmos DB, it can be a convenient place to store offsets.
  • Application State: For simpler scenarios or development, offsets might be stored in memory or a local file, but this is not recommended for production.

When a consumer starts, it retrieves the last known offset for each partition it's interested in from its chosen storage mechanism. It then uses this offset to specify the starting point for reading events from the Event Hub.

Example: Offset Retrieval and Processing (Conceptual)

Consider a scenario where you're reading from partition 0 of an Event Hub. 


// Conceptual C# Example
string lastOffset = GetLastOffsetFromStorage("EventHubName", "ConsumerGroupName", "Partition0");
long startingOffset = string.IsNullOrEmpty(lastOffset) ? 0 : long.Parse(lastOffset);

// Create an Event Hub Receiver starting from the specified offset
EventHubReceiver receiver = eventHubClient.CreateReceiver("ConsumerGroupName", "Partition0", EventPosition.FromOffset(startingOffset));

while (true)
{
    EventData[] events = await receiver.ReceiveAsync(100); // Receive up to 100 events
    if (events.Length > 0)
    {
        foreach (EventData eventData in events)
        {
            // Process the eventData
            Console.WriteLine($"Processing event: {Encoding.UTF8.GetString(eventData.Body.ToArray())}");

            // Update the last processed offset
            lastProcessedOffset = eventData.SequenceNumber; // Event Hubs uses SequenceNumber as offset
        }

        // After successfully processing all events in the batch, commit the last offset
        await SaveLastOffsetToStorage("EventHubName", "ConsumerGroupName", "Partition0", lastProcessedOffset.ToString());
    }
    else
    {
        await Task.Delay(1000); // Wait before trying to receive again
    }
}

Offsets vs. Sequence Numbers

In the context of Azure Event Hubs, the terms offset and sequence number are often used interchangeably. The Event Hubs SDKs and APIs typically refer to these unique, sequential identifiers as sequence numbers. When you retrieve an EventData object, its SequenceNumber property represents the offset of that event within its partition.

Key Takeaway: For practical purposes in Event Hubs, think of SequenceNumber as the offset you need to track.

Best Practices

  • Reliable Storage: Always use a durable and highly available storage solution for your offsets (e.g., Azure Storage, Cosmos DB).
  • Commit After Processing: Commit offsets only after you have successfully processed the corresponding events to enable at-least-once delivery.
  • Handle Consumer Group Lifecycle: Ensure that your offset management strategy accounts for the creation and deletion of consumer groups.
  • Monitor Offset Lag: Monitor the difference between the latest event offset and the offset your consumers are reading from (known as "lag"). High lag can indicate processing bottlenecks.
  • Re-enqueuing: If an event consistently fails processing, consider a strategy to move it aside (e.g., to a "dead-letter" queue) rather than blocking the entire partition.

Further Reading