Upload blobs to Azure Storage

Learn how to efficiently upload data to Azure Blob Storage using various methods.

Azure Blob Storage is a massively scalable object store for unstructured data. This document provides a comprehensive guide to uploading blobs to Azure Blob Storage, covering different scenarios and best practices.

Choosing the Right Upload Method

The best method for uploading blobs depends on your specific requirements, such as the size of the blob, the number of files, and your application's needs. Here are the primary methods:

1. Azure Storage Explorer

Azure Storage Explorer is a cross-platform graphical tool that enables you to easily manage your Azure Storage resources from Windows, macOS, or Linux. It's an excellent choice for manual uploads, managing small to medium-sized files, and exploring your storage account.

Tip: For large, repeated uploads, consider programmatic approaches or AzCopy.

2. AzCopy Command-Line Tool

AzCopy is a command-line utility designed for high-performance data transfer to and from Azure Blob Storage and Azure Files. It's ideal for scripting, automation, and transferring large volumes of data.

Here's a basic example of uploading a file:

azcopy copy "/path/to/my/local/file.txt" "https://myaccount.blob.core.windows.net/mycontainer/myblob.txt[?sv=2020-08-04&ss=bfqt&srt=sco&sp=rwdlacupx&se=2023-12-31T00:00:00Z&st=2023-01-01T00:00:00Z&spr=https&sig=signature]"

For more advanced usage, including syncing directories, specifying storage tiers, and handling network configurations, refer to the AzCopy documentation.

3. Azure SDKs

Azure provides client libraries (SDKs) for various programming languages to interact with Azure Storage programmatically. This is the most flexible and powerful option for integrating blob uploads into your applications.

Uploading a Blob using Azure SDK for .NET

The following code snippet demonstrates how to upload a blob using the Azure Blob Storage SDK for .NET:

// For older SDK versions, ensure you have the Microsoft.Azure.Storage.Blob NuGet package.
// For newer SDK versions, use Azure.Storage.Blobs.

using Azure.Storage.Blobs;
using System;
using System.IO;
using System.Threading.Tasks;

public class BlobUploader
{
    public static async Task UploadBlobAsync(string connectionString, string containerName, string blobName, string localFilePath)
    {
        BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
        BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(containerName);
        await containerClient.CreateIfNotExistsAsync(); // Ensure container exists

        BlobClient blobClient = containerClient.GetBlobClient(blobName);

        using (FileStream uploadFileStream = File.OpenRead(localFilePath))
        {
            await blobClient.UploadAsync(uploadFileStream, overwrite: true);
            Console.WriteLine($"Successfully uploaded blob '{blobName}' to container '{containerName}'.");
        }
    }

    // Example usage:
    // await UploadBlobAsync("YOUR_CONNECTION_STRING", "my-container", "my-blob.txt", "path/to/local/file.txt");
}

Common SDK Operations:

Operation	Description	SDK Method (Conceptual)
Upload Blob	Uploads a new blob or overwrites an existing blob.	`UploadAsync()`
Upload Blocks	Uploads a blob using a series of block uploads. Useful for large files.	`StageBlockAsync(), CommitBlockListAsync()`
Upload Page	Uploads a page blob.	`UploadPagesAsync()`

For other languages, please refer to the respective SDK documentation:

4. REST API

For maximum control or when an SDK is not available, you can use the Azure Blob Storage REST API directly. This involves making HTTP requests to the storage endpoints.

Best Practices for Uploading

Important: For optimal performance and cost-effectiveness, consider the following:

Block Size: For large files, use block blobs and upload them in chunks (blocks). This allows for parallelism and retries.
Parallelism: When using SDKs or AzCopy, leverage multi-threading or parallel operations to speed up uploads.
Storage Tiers: Choose the appropriate access tier (Hot, Cool, Archive) based on how frequently the data will be accessed to optimize costs.
Error Handling: Implement robust error handling and retry mechanisms, especially for network-dependent operations.
Authentication: Use Shared Access Signatures (SAS) or Azure Active Directory (AAD) for secure authentication rather than account keys where possible.

Uploading Large Files

For files larger than 100MB (or up to 5TB for block blobs), it's highly recommended to use the Put Block and Put Block List operations. This is often handled automatically by the SDKs when you use their standard upload methods, but understanding it provides insight into performance tuning.

The process involves:

Staging individual blocks of the blob.
Committing the staged blocks by providing a block list.

Azure Docs