Azure Functions with Azure Event Hubs

A step-by-step guide to processing real-time data streams.

Introduction

This tutorial will guide you through creating an Azure Function that listens to an Azure Event Hub. We'll cover setting up both services, writing the function code, deploying it, and testing the end-to-end flow.

Azure Event Hubs is a highly scalable data streaming platform and event ingestion service. Azure Functions provide an event-driven, serverless compute platform that can be triggered by various Azure services, including Event Hubs.

Prerequisites

If you don't have an Azure account, you can sign up for a free trial.

Setting Up Azure Resources

Before writing any code, we need to provision the necessary Azure resources: an Event Hubs namespace and an Event Hub itself.

1. Create an Event Hubs Namespace and Event Hub

You can do this via the Azure portal or Azure CLI. We'll use the CLI for this tutorial.

1

Log in to your Azure account:

az login
2

Create a resource group (if you don't have one):

az group create --name myEventHubResourceGroup --location eastus
3

Create an Event Hubs namespace:

az eventhubs namespace create --name myEventHubsNamespace --resource-group myEventHubResourceGroup --location eastus
4

Create an Event Hub within the namespace:

az eventhubs eventhub create --name myEventHub --namespace-name myEventHubsNamespace --resource-group myEventHubResourceGroup

2. Get Event Hub Connection String

Your Azure Function will need a connection string to authenticate with the Event Hub. You can retrieve this from the Azure portal or using the CLI.

5

Get the connection string:

az eventhubs authorization-rule list --namespace-name myEventHubsNamespace --resource-group myEventHubResourceGroup --key-type Primary --query "[].{keyName:name, primaryKey:primaryKey}" -o json

Look for the "RootManageSharedAccessKey" rule and copy its primaryKey. You will need to construct the connection string like this:

Endpoint=sb://myEventHubsNamespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=YOUR_PRIMARY_KEY

Important: Replace YOUR_PRIMARY_KEY with the actual primary key you obtained.

Creating the Azure Function

We will create an Azure Function project using Node.js. This function will be triggered by new messages arriving in the Event Hub.

1. Create a New Function App Project

Open your terminal or command prompt and navigate to the directory where you want to create your project.

1

Use the Azure Functions Core Tools to initialize a new project:

func init MyEventHubProcessor --worker-runtime node

Navigate into the project directory:

cd MyEventHubProcessor

2. Add an Event Hub Trigger Function

Now, add a new function that uses the Event Hub trigger.

2

Create the Event Hub trigger function:

func new --name ProcessEventHubMessages --template "Azure Event Hubs trigger"

3. Configure the Function

Open the ProcessEventHubMessages/function.json file and update it to point to your Event Hub. Also, open the local.settings.json file to store your Event Hub connection string securely.

local.settings.json

Add the following to your local.settings.json file. Replace the placeholder with your actual connection string.

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "",
    "FUNCTIONS_WORKER_RUNTIME": "node",
    "EVENTHUB_CONNECTION_STRING": "Endpoint=sb://myEventHubsNamespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=YOUR_PRIMARY_KEY"
  }
}

ProcessEventHubMessages/function.json

Modify the function.json to specify your Event Hub name and the connection setting.

{
  "bindings": [
    {
      "name": "myEventHubMessages",
      "type": "eventHubTrigger",
      "direction": "in",
      "eventHubName": "myEventHub",
      "connection": "EVENTHUB_CONNECTION_STRING",
      "consumerGroup": "$Default",
      "cardinality": "many"
    }
  ],
  "disabled": false
}

4. Write the Function Logic

Open the ProcessEventHubMessages/index.js file and replace its content with the following Node.js code. This function will log the received messages.

module.exports = async function (context, myEventHubMessages) {
    context.log(`JavaScript event hub trigger function processed work item, number of messages: ${myEventHubMessages.length}`);

    myEventHubMessages.forEach((message, index) => {
        context.log(`\tMessage ${index}: ${JSON.stringify(message)}`);
        // You can parse the message content here, e.g., if it's JSON
        try {
            const parsedMessage = JSON.parse(message);
            context.log(`\tParsed Message ${index}:`, parsedMessage);
            // Further processing of parsedMessage
        } catch (e) {
            context.log(`\tCould not parse message ${index} as JSON.`);
        }
    });

    context.log(`Finished processing ${myEventHubMessages.length} messages.`);
};
Note: The consumerGroup in function.json is set to $Default. For production scenarios, it's recommended to create and use dedicated consumer groups to avoid conflicts and ensure reliable message processing.

Deploying the Azure Function

Once your function is ready, you can deploy it to Azure.

1. Create a Function App in Azure

You need a Function App resource in Azure to host your deployed functions.

1

Create a Function App using Azure CLI:

az functionapp create --name myEventHubProcessorApp --resource-group myEventHubResourceGroup --consumption-plan-location eastus --runtime node --os linux --functions-version 4

Replace myEventHubProcessorApp with a unique name for your function app.

2. Deploy the Project

Use the Azure Functions Core Tools to deploy your local project to the created Function App.

2

Deploy your function app:

func azure functionapp publish myEventHubProcessorApp

This command will zip your project and upload it to the specified Function App in Azure.

Testing the Function

To test your function, you need to send messages to the Event Hub.

1. Send a Message to Event Hub

You can use Azure CLI, a C# or Python SDK, or even a simple curl command if you have the necessary authentication setup. Here's an example using Azure CLI with a basic JSON payload.

1

Send a message using Azure CLI (requires `send` permission on the Event Hub, which the RootManageSharedAccessKey has):

az eventhubs send --name myEventHub --namespace-name myEventHubsNamespace --resource-group myEventHubResourceGroup --partition-key "test_key" --body '{"message": "Hello from Azure CLI!"}'

2. Monitor Function Execution

After sending a message, navigate to your Function App in the Azure portal. Go to the "Monitor" section for your ProcessEventHubMessages function.

You should see log entries indicating that your function was triggered and processed the message. The logs will show the message content that was sent.

If you encounter issues, check the "Diagnose and solve problems" blade within your Function App for common troubleshooting steps.

Further Enhancements