Blob Lifecycle Management

This document provides an in-depth guide to configuring and managing blob lifecycle policies in Azure Storage.

Introduction

Azure Blob Storage lifecycle management allows you to automate the transition of blob data to the most appropriate, cost-effective access tier and to expire or delete blobs at the end of their lifecycle. This is crucial for optimizing costs and managing data according to compliance and business requirements.

You can use lifecycle management to:

• Transition blobs from cool to archive tier to save costs.
• Transition blobs from hot to cool tier to save costs.
• Delete blobs from the subscription when they are no longer needed.
• Transition blobs based on their creation date or last modified date.
• Apply rules to all blobs in a container or a subset of blobs filtered by a prefix.

How Lifecycle Management Works

Lifecycle management policies are defined as a set of rules. Each rule consists of a scope, filters, and actions. The Azure Storage service evaluates these rules periodically and applies the defined actions to blobs that match the specified filters.

The policy applies to blobs within a storage account. You can apply a single policy to multiple storage accounts. Rules within a policy are evaluated independently.

Rule Scope

Rules can be applied to an entire storage account or to a subset of blobs defined by filters.

Rule Filters

Filters allow you to target specific blobs for lifecycle management. You can filter blobs based on:

• Blob type: Block blobs only.
• Prefix matching: Apply rules to blobs with specific prefixes (e.g., /logs/ or /archive/data).
• Creation date: Blobs created before a certain date.
• Last modification date: Blobs last modified before a certain date.
• Last access date: Blobs last accessed before a certain date.

Rule Actions

Actions define what happens to the blobs that match the rule's filters. Available actions include:

• Transition: Move blobs to a different access tier (e.g., Hot to Cool, Cool to Archive).
• Delete: Delete blobs.

Actions can be scheduled relative to the blob's creation, modification, or access date.

Creating a Lifecycle Policy

You can create lifecycle management policies using the Azure portal, Azure CLI, or Azure PowerShell.

Using the Azure Portal

1. Navigate to your storage account in the Azure portal.
2. Under the "Data management" section, select "Lifecycle management".
3. Click "Add a rule".
4. Configure the rule name, scope, filters, and actions.
5. Save the policy.

Using Azure CLI

Use the az storage lifecycle rule create command:

az storage lifecycle rule create \
    --account-name  \
    --resource-group  \
    --name 'transition-to-cool' \
    --blob-type BlockBlob \
    --rule-type Management \
    --actions '{"type":"Delete","daysAfterModificationGreaterThan":365}' \
    --filters '{"prefixMatch":["archive/"]}' \
    --enable true
        

Using Azure PowerShell

Use the New-AzStorageAccountManagementPolicyRule cmdlet:

$action = New-AzStorageAccountManagementPolicyAction -BaseBlobActionType Delete -DaysAfterModificationGreaterThan 365
$filter = New-AzStorageAccountManagementPolicyFilter -BlobType 'BlockBlob' -Prefix "archive/"
$rule = New-AzStorageAccountManagementPolicyRule -Name "DeleteOldArchives" -Action $action -Filter $filter
Set-AzStorageAccountManagementPolicy -ResourceGroupName "" -StorageAccountName "" -Policy $rule
        

Examples

Example 1: Transitioning Hot to Cool and then to Archive

This policy transitions blobs to the cool tier after 30 days and then to the archive tier after 90 days.

{
    "policy": {
        "rules": [
            {
                "name": "transition-hot-to-cool-and-archive",
                "enabled": true,
                "type": "Lifecycle",
                "definition": {
                    "actions": {
                        "versioning": null,
                        "snapshot": null,
                        "baseBlob": {
                            "type": "Move",
                            "daysAfterModificationGreaterThan": 30,
                            "daysAfterLastAccessGreaterThan": null,
                            "tier": "Cool"
                        },
                        "container": null
                    },
                    "filters": {
                        "blobTypes": [
                            "BlockBlob"
                        ],
                        "blobPrefixOffsets": null,
                        "matchBlobPrefix": null,
                        "matchBlobTags": null,
                        "daysAfterCreationGreaterThan": null,
                        "daysAfterModificationGreaterThan": null,
                        "daysAfterLastAccessGreaterThan": null
                    }
                }
            },
            {
                "name": "transition-cool-to-archive",
                "enabled": true,
                "type": "Lifecycle",
                "definition": {
                    "actions": {
                        "versioning": null,
                        "snapshot": null,
                        "baseBlob": {
                            "type": "Move",
                            "daysAfterModificationGreaterThan": 90,
                            "daysAfterLastAccessGreaterThan": null,
                            "tier": "Archive"
                        },
                        "container": null
                    },
                    "filters": {
                        "blobTypes": [
                            "BlockBlob"
                        ],
                        "blobPrefixOffsets": null,
                        "matchBlobPrefix": null,
                        "matchBlobTags": null,
                        "daysAfterCreationGreaterThan": null,
                        "daysAfterModificationGreaterThan": null,
                        "daysAfterLastAccessGreaterThan": null
                    }
                }
            }
        ]
    }
}
        

Example 2: Deleting Blobs After a Year

This policy deletes blobs that haven't been modified in over 365 days.

{
    "policy": {
        "rules": [
            {
                "name": "delete-old-blobs",
                "enabled": true,
                "type": "Lifecycle",
                "definition": {
                    "actions": {
                        "versioning": null,
                        "snapshot": null,
                        "baseBlob": {
                            "type": "Delete",
                            "daysAfterModificationGreaterThan": 365,
                            "daysAfterLastAccessGreaterThan": null
                        },
                        "container": null
                    },
                    "filters": {
                        "blobTypes": [
                            "BlockBlob"
                        ],
                        "blobPrefixOffsets": null,
                        "matchBlobPrefix": null,
                        "matchBlobTags": null,
                        "daysAfterCreationGreaterThan": null,
                        "daysAfterModificationGreaterThan": null,
                        "daysAfterLastAccessGreaterThan": null
                    }
                }
            }
        ]
    }
}
        

Best Practices

• Start simple: Begin with a few basic rules and iterate as you understand your data access patterns.
• Use prefixes effectively: Organize your blobs with prefixes to create granular lifecycle policies.
• Monitor costs: Regularly review your storage costs to ensure your lifecycle policies are optimizing expenditure.
• Test rules: Apply rules to a small subset of data first to verify their behavior before rolling them out broadly.
• Consider rehydration costs: If you frequently access data in the archive tier, factor in the rehydration costs.
• Use versioning: If you are using blob versioning, configure lifecycle management rules to manage older versions as well.