Azure Blob Storage Lifecycle Management

Overview

Azure Blob Storage lifecycle management allows you to manage the lifecycle of your blobs by applying rules that transition blobs to different access tiers or delete them at the end of their lifecycle. This helps you optimize costs by moving less frequently accessed data to cooler tiers and by deleting data that is no longer needed.

Lifecycle management policies are applied at the storage account level. A policy consists of a set of rules. Each rule targets a subset of blobs within the storage account and defines actions to be taken based on the blobs' age or other criteria.

Key Concepts

• Rules: A set of conditions and actions. Rules can be applied to blobs based on filters like prefix, blob type, or tags.
• Filters: Criteria used to select blobs for a rule. Common filters include blob prefix, blob type (block, append, page), and tags.
• Actions: Operations to perform on the filtered blobs. These include transitioning blobs to a different access tier (Hot, Cool, Archive) or deleting blobs.
• Access Tiers:
  • Hot: For frequently accessed data.
  • Cool: For infrequently accessed data, optimized for cost savings.
  • Archive: For rarely accessed data, optimized for the lowest storage cost. Retrieval times can be longer.
• Age: The number of days since a blob was last accessed or created.

Creating a Lifecycle Management Policy

You can create and manage lifecycle management policies using the Azure portal, Azure CLI, or Azure PowerShell.

Using Azure Portal:

1. Navigate to your storage account in the Azure portal.
2. In the left-hand menu, under "Data management", select "Lifecycle management".
3. Click on the "+ Add a rule" button.
4. Configure the rule's name, scope (account level), and filters.
5. Define the actions, such as transitioning to Cool tier after 30 days or deleting after 365 days.
6. Review and save the rule.

Using Azure CLI:

Here's an example command to create a policy that moves blobs to the Cool tier after 30 days and deletes them after 365 days.

az storage lifecycle-management policy create \
    --account-name <your-storage-account-name> \
    --resource-group <your-resource-group-name> \
    --policy '{
        "rules": [
            {
                "name": "transition_to_cool_and_delete",
                "enabled": true,
                "type": "Management",
                "definition": {
                    "filters": {
                        "blobTypes": ["blockBlob"]
                    },
                    "actions": {
                        "baseBlob": {
                            "tierToCool": {
                                "daysAfterModificationGreaterThan": 30
                            },
                            "delete": {
                                "daysAfterModificationGreaterThan": 365
                            }
                        }
                    }
                }
            }
        ]
    }'
                

Policy Definition Structure:

A lifecycle policy is defined as a JSON object containing an array of rules.

{
    "rules": [
        {
            "name": "string",
            "enabled": boolean,
            "type": "Management", // Can be "Management" or "Action"
            "definition": {
                "filters": {
                    "blobTypes": ["blockBlob", "appendBlob", "pageBlob"],
                    "prefixMatch": ["string"],
                    "tags": {
                        "key": "value"
                    }
                },
                "actions": {
                    "baseBlob": {
                        "tierToCool": { "daysAfterModificationGreaterThan": integer },
                        "tierToArchive": { "daysAfterModificationGreaterThan": integer },
                        "delete": { "daysAfterModificationGreaterThan": integer }
                    },
                    "snapshot": {
                        "tierToCool": { "daysAfterSnapshotGreaterThan": integer },
                        "tierToArchive": { "daysAfterSnapshotGreaterThan": integer },
                        "delete": { "daysAfterSnapshotGreaterThan": integer }
                    },
                    "version": {
                        "tierToCool": { "daysAfterCreationGreaterThan": integer },
                        "tierToArchive": { "daysAfterCreationGreaterThan": integer },
                        "delete": { "daysAfterCreationGreaterThan": integer }
                    }
                }
            }
        }
    ]
}
                

Rule Examples

{
    "name": "transition_cool_archive",
    "enabled": true,
    "type": "Management",
    "definition": {
        "filters": {
            "blobTypes": ["blockBlob"],
            "prefixMatch": ["logs/"]
        },
        "actions": {
            "baseBlob": {
                "tierToCool": { "daysAfterModificationGreaterThan": 30 },
                "tierToArchive": { "daysAfterModificationGreaterThan": 90 }
            }
        }
    }
}
                    

{
    "name": "delete_old_blobs",
    "enabled": true,
    "type": "Management",
    "definition": {
        "filters": {
            "blobTypes": ["blockBlob"]
        },
        "actions": {
            "baseBlob": {
                "delete": { "daysAfterModificationGreaterThan": 365 }
            }
        }
    }
}
                    

{
    "name": "tag_based_transition",
    "enabled": true,
    "type": "Management",
    "definition": {
        "filters": {
            "blobTypes": ["blockBlob"],
            "tags": {
                "environment": "development"
            }
        },
        "actions": {
            "baseBlob": {
                "tierToCool": { "daysAfterModificationGreaterThan": 15 }
            }
        }
    }
}
                    

Best Practices and Considerations

• Test thoroughly: Before applying policies to production data, test them in a development or staging environment.
• Understand costs: Each tier has different pricing for storage, transactions, and data retrieval. Archive retrieval can incur significant costs and time.
• Use prefixes and tags: Effectively use filters to target specific groups of blobs, making your policies more precise and manageable.
• Monitor: Regularly review the execution logs of your lifecycle management policies to ensure they are working as expected.
• Immutable Blobs: Lifecycle management policies do not apply to immutable blobs.
• Soft Delete: Ensure your soft delete policies are configured appropriately, as they interact with lifecycle management.

Troubleshooting