Azure Storage Blobs

Key Concepts for Large File Uploads

When dealing with large files (typically over 100MB), it's crucial to consider:

• Block Blobs: The default blob type, ideal for storing large amounts of unstructured data like files, images, videos, and backups. Large files are uploaded as a series of blocks.
• Chunking: Breaking down a large file into smaller, manageable chunks for uploading.
• Parallel Uploads: Uploading multiple chunks concurrently to significantly reduce upload time.
• Resumability: The ability to resume an interrupted upload without starting from scratch.
• Throttling: Azure Storage has request rate limits. Efficient chunking and parallelization help manage these.

1. Using Azure SDKs (Recommended)

The Azure SDKs provide high-level abstractions that handle many of the complexities of large file uploads, including chunking, parallel uploads, and retry mechanisms.

Here's a conceptual example using the Python SDK:

from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient
import os

# Replace with your actual connection string and container name
connect_str = os.getenv("AZURE_STORAGE_CONNECTION_STRING")
container_name = "my-upload-container"
local_file_name = "path/to/your/large/file.zip"
blob_name = "my-large-upload-file.zip"

def upload_large_blob(local_path, blob_service_client, container_name, blob_name):
    try:
        container_client = blob_service_client.get_container_client(container_name)
        blob_client = container_client.get_blob_client(blob_name)

        print(f"Uploading {local_path} to {blob_name}...")
        with open(local_path, "rb") as data:
            blob_client.upload_blob(data, overwrite=True)
        print("Upload complete!")

    except Exception as e:
        print(f"An error occurred: {e}")

if __name__ == "__main__":
    if not connect_str:
        print("Please set the AZURE_STORAGE_CONNECTION_STRING environment variable.")
    else:
        blob_service_client = BlobServiceClient.from_connection_string(connect_str)
        upload_large_blob(local_file_name, blob_service_client, container_name, blob_name)
                        

Similar implementations are available in .NET, Java, Node.js, Go, and other languages.

2. Using AzCopy

AzCopy is a command-line utility designed for high-performance data transfer to and from Azure Blob Storage and File Storage. It's highly optimized for large files and supports features like resuming uploads.

Command Example:

azcopy copy "C:\path\to\your\large\file.zip" "https://.blob.core.windows.net//my-large-upload-file.zip?" --recursive=true
                

AzCopy automatically handles chunking and parallel uploads for optimal performance.

3. Using Azure CLI

The Azure CLI also provides commands for interacting with Azure Storage. The az storage blob upload-batch command can be used for uploading directories, and it also handles large files efficiently.

Command Example:

az storage blob upload \
    --account-name  \
    --container-name  \
    --name my-large-upload-file.zip \
    --file C:\path\to\your\large\file.zip \
    --auth-mode login
                

The CLI leverages underlying libraries that manage large file uploads.

4. Using the REST API (Advanced)

For maximum control or integration into custom applications, you can use the Azure Storage REST API directly. This involves initiating a block blob upload, uploading blocks, and then committing the blob.

Key Operations:

• Put Block List: Commits the uploaded blocks to form the blob.
• Put Block: Uploads a single block of data.

This method requires manual implementation of chunking, block IDs, and managing HTTP requests.