Azure Data Lake Storage CLI Reference

Introduction

The Azure Command-Line Interface (CLI) provides a set of commands for managing Azure resources. This document focuses on the CLI commands specifically for interacting with Azure Data Lake Storage Gen2. You can perform operations such as creating file systems, uploading files, setting permissions, and more, directly from your terminal.

Prerequisites

Before you can use the Azure CLI with Data Lake Storage Gen2, you need:

An Azure subscription.
An Azure Data Lake Storage Gen2 enabled storage account.
The Azure CLI installed on your local machine or available in Azure Cloud Shell.

Installation

To install the Azure CLI, follow the official instructions for your operating system:

Note: It is recommended to install the latest version of the Azure CLI for the best experience and latest features.

You can typically install it using package managers or by downloading an installer.

For example, on most Linux distributions:

curl -sL https://aka.ms/InstallAzureCLIDebian | sudo bash

Or using npm:

npm install -g azure-cli

Refer to the official Azure CLI installation guide for detailed steps.

Login to Azure

Before executing any commands, you must log in to your Azure account.

az login

This command will open a browser window for you to authenticate. If you are using Azure Cloud Shell, you are likely already logged in.

To specify a different subscription:

az account set --subscription <Your-Subscription-ID>

Storage Account Management

Manage your Azure Storage accounts, including Data Lake Storage Gen2 enabled accounts.

Create a Storage Account

To create a new storage account that can be used for Data Lake Storage Gen2:

az storage account create \
    --name <your-storage-account-name> \
    --resource-group <your-resource-group-name> \
    --location <azure-region> \
    --kind StorageV2 \
    --sku Standard_LRS \
    --hierarchical-namespace true

Replace placeholders like <your-storage-account-name>, <your-resource-group-name>, and <azure-region> with your specific values. Setting --hierarchical-namespace true is crucial for Data Lake Storage Gen2.

List Storage Accounts

az storage account list --resource-group <your-resource-group-name>

Filesystem Operations

Manage containers (filesystems) within your Data Lake Storage Gen2 account.

Create a Filesystem

Use the az storage fs create command to create a new filesystem.

az storage fs create \
    --name <your-filesystem-name> \
    --account-name <your-storage-account-name> \
    --account-key <your-storage-account-key>

Alternatively, if you have logged in and set the context:

az storage fs create \
    --name <your-filesystem-name> \
    --account-name <your-storage-account-name>

You can also authenticate using a connection string or a managed identity.

List Filesystems

az storage fs list \
    --account-name <your-storage-account-name>

Delete a Filesystem

az storage fs delete \
    --name <your-filesystem-name> \
    --account-name <your-storage-account-name>

File and Directory Operations

Perform operations on files and directories within a filesystem.

Upload a File

Upload a local file to a directory in your filesystem.

az storage fs file upload \
    --file <path-to-local-file> \
    --destination-path <destination-directory-in-fs> \
    --fs-name <your-filesystem-name> \
    --account-name <your-storage-account-name>

To upload to the root of the filesystem, use --destination-path "/" or omit it if it defaults correctly.

Download a File

az storage fs file download \
    --file <path-to-file-in-fs> \
    --destination-path <path-to-local-destination> \
    --fs-name <your-filesystem-name> \
    --account-name <your-storage-account-name>

List Files and Directories

List the contents of a directory within a filesystem.

az storage fs directory list \
    --path <directory-in-fs> \
    --fs-name <your-filesystem-name> \
    --account-name <your-storage-account-name>

Create a Directory

az storage fs directory create \
    --name <new-directory-name> \
    --path <parent-directory-in-fs> \
    --fs-name <your-filesystem-name> \
    --account-name <your-storage-account-name>

Access Control and Permissions

Azure Data Lake Storage Gen2 uses Access Control Lists (ACLs) for fine-grained permissions.

Set ACLs

You can set ACLs for files and directories.

az storage fs access set \
    --acl "user:<user-principal-name>:rwx" \
    --path <path-to-file-or-directory> \
    --fs-name <your-filesystem-name> \
    --account-name <your-storage-account-name>

Permissions can be read (r), write (w), and execute (x).



                Get ACLs
                
                    az storage fs access show \
    --path <path-to-file-or-directory> \
    --fs-name <your-filesystem-name> \
    --account-name <your-storage-account-name>