Azure Documentation

Accessing Azure Data Lake Storage Gen2

Azure Data Lake Storage Gen2 is built on Azure Blob Storage, offering a hierarchical namespace. This allows for efficient data access and management, similar to a file system. Understanding how to access your data is crucial for leveraging its full potential.

Methods of Access

You can access your Data Lake Storage Gen2 data through various methods, catering to different use cases and technical preferences:

  • Azure Portal: The graphical user interface for managing Azure resources.
  • Azure CLI: A command-line tool for managing Azure resources.
  • Azure PowerShell: A scripting language and command-line shell for managing Azure resources.
  • Azure Storage SDKs: Libraries for various programming languages (e.g., Python, Java, .NET, Node.js) that provide programmatic access.
  • REST API: Direct HTTP requests to interact with the storage service.
  • Third-Party Tools: Various data exploration and management tools that integrate with Azure Data Lake Storage Gen2.

Accessing via Azure Portal

The Azure portal provides a user-friendly way to browse, upload, download, and manage your data. Navigate to your storage account, then select "Containers" under "Data storage". You can then browse your hierarchical namespace directly.

Accessing via Azure CLI

The Azure CLI is a powerful tool for scripting and automation. To access Data Lake Storage Gen2, you'll primarily use commands related to Azure Storage. For example, to list directories in a container:


az storage fs list --account-name  --auth-mode login
az storage fs directory list --account-name  --file-system  --auth-mode login

To download a file:


az storage fs file download --account-name  --file-system  --path  --destination  --auth-mode login

Accessing via Azure Storage SDKs

Using the SDKs offers the most flexibility for integrating data access into your applications. Here's a conceptual example using Python:


from azure.storage.blob import BlobServiceClient

connection_string = "YOUR_AZURE_STORAGE_CONNECTION_STRING"
blob_service_client = BlobServiceClient.from_connection_string(connection_string)
file_system_client = blob_service_client.get_file_system_client(file_system="your-container-name")

# List directories and files
paths = file_system_client.list_paths("your/directory")
for path in paths:
    print(path.name)

# Download a file
blob_client = file_system_client.get_blob_client(blob=f"your/directory/your_file.txt")
with open("downloaded_file.txt", "wb") as download_file:
    download_file.write(blob_client.download_blob().readall())
Tip: For programmatic access, consider using Azure Identity libraries for secure authentication without hardcoding credentials.

Authentication and Authorization

Secure access is paramount. Data Lake Storage Gen2 supports several authentication methods:

  • Azure Active Directory (Azure AD): Recommended for most scenarios, providing robust identity and access management.
  • Shared Access Signatures (SAS): Time-limited, scoped access tokens.
  • Access Keys: Direct credentials to the storage account (use with caution).

Authorization is managed through Azure Role-Based Access Control (RBAC) and Access Control Lists (ACLs) for granular permissions on directories and files within the hierarchical namespace.

Key Concepts

  • Hierarchical Namespace: Organizes data into directories and subdirectories.
  • Containers: The top-level organizational unit for your data.
  • Blobs: The actual data files stored within the Data Lake.

By understanding these access methods and concepts, you can effectively integrate Azure Data Lake Storage Gen2 into your data pipelines and analytical workloads.