Retention Policy

Understanding and configuring the data retention policy for your Azure Event Hubs is crucial for managing storage costs, compliance requirements, and the lifecycle of your event data.

What is Data Retention?

Data retention refers to the duration for which events are stored in an Event Hub. Once the retention period expires, events are automatically deleted. Azure Event Hubs supports configuring this policy at the namespace level, which then applies to all Event Hubs within that namespace.

Configuring Retention Policy

The retention period can be configured in hours or days. The default retention period is 24 hours. You can set a retention period of up to 7 days for standard tiers and up to 30 days for premium and dedicated tiers.

Using Azure Portal

1. Navigate to your Event Hubs namespace in the Azure portal.

2. Under the "Settings" section, select "Event Hubs".

3. Click on the specific Event Hub you want to configure (or set it at the namespace level if desired).

4. Look for the "Message Retention" setting. You can specify the duration in hours or days.

5. Click "Save" to apply the changes.

Using Azure CLI

You can use the Azure CLI to manage Event Hubs, including setting the retention policy.

az eventhubs eventhub update --resource-group  --namespace-name  --name  --message-retention

Replace <YourResourceGroup>, <YourNamespaceName>, <YourEventHubName>, and <RetentionInHours> with your specific values. The retention value is in hours.

Using Azure SDKs

When creating or updating an Event Hub using the Azure SDKs (e.g., Python, .NET, Java), you can specify the retention period as a property of the Event Hub configuration.

Important: Changing the retention policy affects all Event Hubs within the namespace if configured at the namespace level. For specific Event Hubs, configure it directly on the Event Hub.

Considerations for Retention Policy

  • Storage Costs: Longer retention periods mean more data stored, which can increase costs.
  • Compliance: Some industries have regulatory requirements for data retention. Ensure your policy meets these needs.
  • Processing Lag: If your consumers have a significant processing lag, ensure the retention period is long enough to avoid data loss.
  • Archival: For long-term archival, consider integrating with Azure Blob Storage or Azure Data Lake Storage. Event Hubs Capture can automatically archive data to these services.
Data Loss: Once data is expired based on the retention policy, it is permanently deleted and cannot be recovered. Plan your retention periods carefully.

Event Hubs Capture and Retention

Event Hubs Capture is a built-in feature that automatically and incrementally archives Event Hubs data to a specified Azure Blob Storage account or Azure Data Lake Storage account. When using Capture, the retention policy of the Event Hub itself is still relevant for active data in the hub, while Capture manages the archival storage independently.

Further Reading