Microsoft Azure Community

Hi everyone, I'm encountering a significant performance bottleneck when uploading large files (several GBs) to Azure Blob Storage from a geographically distant location with high network latency. The upload speeds are much lower than expected, even when using parallel uploads. We are using the Azure Blob Storage SDK for Python, specifically targeting `BlobClient.upload_blob`. We've experimented with `max_block_size` and `max_concurrency`, but haven't seen a substantial improvement. Are there any specific configurations or best practices for optimizing blob uploads in high-latency environments? We've checked our network connection, and it's stable, but the latency is inherent to the geographical distance. Any insights or suggestions would be greatly appreciated!

Hello John, I understand your frustration. High latency can indeed be a challenge. Have you considered using Azure Data Factory or AzCopy for your large file transfers? These tools are often optimized for large-scale data movement and can handle network inefficiencies better. Also, ensure you're uploading to a blob storage account located in the same region as your client or as close as possible to minimize latency from the start.

Hi Alice, Thanks for the suggestions. We are indeed uploading to a storage account in the closest Azure region. We haven't extensively tested AzCopy yet, but we will definitely try it out. Our current workflow requires programmatic uploads directly from our application, so relying solely on a separate tool like ADF might not be ideal for every scenario, but we'll consider it for bulk operations. Are there any SDK-specific parameters related to TCP window scaling or connection pooling that we might be overlooking?