Azure Event Hubs - Capture Feature

Event Hubs Capture

Azure Event Hubs Capture is a built-in feature that automatically and incrementally batches the output of Event Hubs into the storage solution of your choice. It's designed for scenarios where you need to reliably capture streaming data for long-term archival, batch analytics, or immediate reprocessing.

Effortlessly archive your event streams to Azure Blob Storage or Azure Data Lake Storage.

How it Works

When Capture is enabled, Event Hubs automatically takes snapshots of the data and writes them to a storage account. This process is managed by Azure, requiring minimal configuration on your part. The data is written in Apache Avro format, which is efficient and widely supported by big data analytics platforms.

Key Benefits

Fully Managed: No infrastructure to manage, just configure and forget.
Scalable: Scales automatically with your Event Hubs throughput.
Reliable: Guarantees that all events are captured.
Flexible: Supports both Azure Blob Storage and Azure Data Lake Storage Gen2.
Cost-Effective: Pay only for the storage consumed.
Avro Format: Optimized for batch processing and analytics.

Use Cases

Archival: Store event data for compliance, auditing, or future analysis.
Batch Analytics: Feed data directly into services like Azure Databricks, Azure Synapse Analytics, or Hadoop for batch processing.
Reprocessing: Provide a replayable dataset for debugging or to recover from processing errors.
Data Exploration: Enable data scientists and analysts to explore raw event data.

Enabling Capture

You can enable Capture through the Azure portal, Azure CLI, or Azure Resource Manager (ARM) templates.

Azure Portal Steps:

Navigate to your Event Hubs namespace in the Azure portal.
In the left-hand menu, select "Capture" under "Settings".
Toggle "Capture" to "On".
Choose your desired storage account (Blob Storage or Data Lake Storage Gen2).
Select or create a container for the captured data.
Configure the capture window (time and size) for batching.
Click "Save".

Example Azure CLI Command:


az eventhubs eventhub update --resource-group <your-resource-group> \
    --namespace-name <your-namespace> \
    --name <your-eventhub-name> \
    --capture-enabled \
    --capture-interval 60 \
    --capture-size 104857600 \
    --capture-destination blob \
    --capture-blob-container <your-container-name> \
    --capture-storage-account <your-storage-account-name>

Capture Configuration Options

Capture Interval

Specifies the maximum time interval (in seconds) between captures. Data will be captured at least this often.

Capture Size Limit

Specifies the maximum size (in bytes) of captured data before it's written to storage.

Destination Type

Choose between Azure Blob Storage or Azure Data Lake Storage Gen2.

Storage Account & Container

The specific Azure Storage account and container where captured data will be stored.

File Naming Convention

Event Hubs Capture generates files with a structured naming convention, including namespace, event hub name, partition ID, and timestamps, making it easy to organize and query.

Learn More

For more in-depth information and advanced configurations, please refer to the official Azure Event Hubs documentation: