Azure Data Lake Storage Gen2

Introduction to the Azure Data Lake Storage Gen2 JavaScript SDK

The Azure Data Lake Storage Gen2 JavaScript SDK provides a robust and efficient way to interact with your Data Lake Storage Gen2 accounts from your Node.js and browser-based applications. This SDK leverages the Azure Storage REST API to offer functionalities for managing file systems, directories, and files.

Key features include:

• Seamless integration with Azure Active Directory (AAD) for authentication.
• High-level abstractions for common data lake operations.
• Support for streaming large files.
• Comprehensive error handling and retry policies.

Installation

You can install the Azure Data Lake Storage Gen2 SDK using npm or yarn:

npm install @azure/storage-file-datalake
# or
yarn add @azure/storage-file-datalake
                

Getting Started

To get started, you'll need an Azure Data Lake Storage Gen2 account and a connection string or credentials. Here's a basic example of how to create a DataLakeServiceClient:

import { DataLakeServiceClient } from "@azure/storage-file-datalake";
import { DefaultAzureCredential } from "@azure/identity";

async function main() {
    const accountName = "YOUR_STORAGE_ACCOUNT_NAME";
    const credential = new DefaultAzureCredential(); // Or use other credentials like SharedTokenCacheCredential

    const serviceClient = new DataLakeServiceClient(
        `https://${accountName}.dfs.core.windows.net`,
        credential
    );

    console.log("DataLakeServiceClient created successfully.");

    // Now you can use serviceClient to manage file systems
    // For example:
    // const fileSystems = serviceClient.listFileSystems();
    // for await (const fs of fileSystems) {
    //     console.log(fs.name);
    // }
}

main().catch((error) => {
    console.error("The following error occurred: ", error);
    process.exit(1);
});
                

Replace YOUR_STORAGE_ACCOUNT_NAME with your actual storage account name. The DefaultAzureCredential automatically tries to authenticate using environment variables, managed identity, or other available methods.

DataLakeFileSystemClient

The DataLakeFileSystemClient represents a file system within your Data Lake Storage Gen2 account. You can obtain an instance of this client from the DataLakeServiceClient.

Creating a File System Client

// Assuming serviceClient is already initialized as shown in Getting Started

const fileSystemName = "myfilesystem";
const fileSystemClient = serviceClient.getFileSystemClient(fileSystemName);
console.log(`FileSystemClient for '${fileSystemName}' created.`);
                

createFileSystem(fileSystemName: string, options?: FileSystemCreateOptionalParams)

Creates a new file system within the storage account.

Parameters:

Returns:

deleteFileSystem(options?: FileSystemDeleteOptionalParams)

Deletes the file system.

Parameters:

Returns:

listFileSystems(options?: FileSystemsListOptionalParams)

Lists all file systems in the storage account.

Parameters:

Returns:

getFileSystem(options?: FileSystemGetOptionalParams)

Gets properties of the file system.

Parameters:

Returns:

DataLakeDirectoryClient

The DataLakeDirectoryClient represents a directory within a file system. You can obtain an instance of this client from the DataLakeFileSystemClient.

Creating a Directory Client

// Assuming fileSystemClient is already initialized

const directoryName = "mydata/raw";
const directoryClient = fileSystemClient.getDirectoryClient(directoryName);
console.log(`DirectoryClient for '${directoryName}' created.`);
                

createDirectory(options?: DirectoryCreateOptionalParams)

Creates a new directory.

Parameters:

Returns:

deleteDirectory(options?: DirectoryDeleteOptionalParams)

Deletes the directory. The directory must be empty.

Parameters:

Returns:

listPaths(options?: PathsListOptionalParams)

Lists the paths (files and subdirectories) within this directory.

Parameters:

Returns:

DataLakeFileClient

The DataLakeFileClient represents a file within a directory or file system. You can obtain an instance of this client from the DataLakeFileSystemClient or DataLakeDirectoryClient.

Creating a File Client

// Assuming fileSystemClient is already initialized

const fileName = "mydata/raw/sales.csv";
const fileClient = fileSystemClient.getFileClient(fileName);
console.log(`FileClient for '${fileName}' created.`);
                

createFile(options?: FileCreateOptionalParams)

Creates a new file. If the file already exists, this operation overwrites it.

Parameters:

Returns:

deleteFile(options?: FileDeleteOptionalParams)

Deletes the file.

Parameters:

Returns:

readFile(options?: FileReadOptionalParams)

Reads the content of the file. Returns a readable stream.

Parameters:

Returns:

Example of reading a file:

import { pipeline } from "stream/promises";
import { createWriteStream } from "fs";

// Assuming fileClient is initialized for a file

async function downloadFile(fileClient, localPath) {
    const response = await fileClient.readFile();
    const destination = createWriteStream(localPath);
    await pipeline(response.readableStreamBody, destination);
    console.log(`File downloaded to ${localPath}`);
}

// downloadFile(fileClient, "./local_sales.csv").catch(console.error);
                

appendData(data: Uint8Array | ArrayBuffer | Buffer, options?: FileAppendDataOptionalParams)

Appends data to the end of the file. This operation is asynchronous and may require multiple calls for large data chunks.

Parameters:

Returns:

flushData(options?: FileFlushDataOptionalParams)

Commits the data that has been appended to the file. This operation finalizes the file's content up to the specified length.

Parameters:

Returns:

Example of writing to a file:

// Assuming fileClient is initialized

async function writeFileContent(fileClient, content) {
    // Create the file if it doesn't exist (overwrites if it does)
    await fileClient.createFile();

    // Append data
    await fileClient.appendData(Buffer.from(content));

    // Flush data to commit
    await fileClient.flushData({
        finalSize: Buffer.byteLength(content)
    });
    console.log("File content written successfully.");
}

// writeFileContent(fileClient, "This is the content of my file.").catch(console.error);
                

getFileProperties(options?: FileGetPropertiesOptionalParams)

Gets the properties of the file.

Parameters:

Returns: