Azure Data Factory Pipeline Expressions

Pipeline Expressions in Azure Data Factory

Pipeline expressions allow you to dynamically construct strings at runtime. You can use them to set property values, pass parameters, and control pipeline logic. Azure Data Factory supports a rich set of functions that can be used within expressions.

Overview

Expressions are string literals with an expression body enclosed in curly braces { }. The expression body can contain a combination of:

System variables (e.g., @pipeline().TriggerTime)
Parameters (e.g., @pipeline().parameters.MyParameter)
Activity output (e.g., @activity('MyActivity').output.MyOutputProperty)
Functions (e.g., @concat(), @formatDateTime())
Static values (e.g., string literals, numbers)

Common Functions

String Functions

@concat(str1, str2, ...): Concatenates multiple strings.
@string(value): Converts a value to a string.
@length(str): Returns the length of a string.
@substring(str, startIndex, length): Extracts a substring.
@toLower(str): Converts a string to lowercase.
@toUpper(str): Converts a string to uppercase.

Date and Time Functions

@utcNow(): Returns the current UTC date and time.
@formatDateTime(dateTime, format): Formats a date and time value.
@addHours(dateTime, hours): Adds hours to a date and time value.
@addMinutes(dateTime, minutes): Adds minutes to a date and time value.

Array Functions

@length(array): Returns the number of elements in an array.
@first(array): Returns the first element of an array.
@last(array): Returns the last element of an array.

Collection Functions

@contains(collection, item): Checks if a collection contains a specific item.

Conditional Functions

@if(condition, trueValue, falseValue): Returns one of two values based on a condition.

System Variables

System variables provide context about the pipeline run. Some common ones include:

@pipeline().RunId: The unique identifier for the current pipeline run.
@pipeline().TriggerTime: The timestamp when the pipeline was triggered.
@pipeline().PipelineRunStart: The start time of the current pipeline run.
@pipeline().parameters.: Accesses a pipeline parameter.
@activity('').output.: Accesses the output of a previous activity.

Examples

Concatenating strings and system variables

@concat('This is pipeline run ID: ', @pipeline().RunId, ' triggered at ', @formatDateTime(pipeline().TriggerTime, 'yyyy-MM-dd HH:mm:ss'))

Using parameters and conditional logic

@if(equals(pipeline().parameters.Environment, 'Production'), 'output/prod', 'output/dev')

Accessing activity output

@activity('CopyDataActivity').output.copy.source.effectiveIntegrationRuntime

Constructing a file path with a date

@concat('logs/', @formatDateTime(utcNow(), 'yyyy/MM/dd'), '/activity.log')

Best Practices

Use clear and descriptive parameter names.
Leverage built-in functions to avoid complex custom logic.
Test your expressions thoroughly in a development environment.
Be mindful of casing for function names and system variables.