Concepts of Azure Data Lake Storage Gen2

In this article: Introduction| Core Concepts| Hierarchical Namespace| Azure Blob Storage Integration| Security and Access Control| Performance and Scalability

Introduction

Azure Data Lake Storage Gen2 (ADLS Gen2) is a powerful, scalable, and secure data lake solution built on Azure Blob Storage. It is designed for big data analytics workloads, offering a hierarchical namespace that provides capabilities similar to a Hadoop Distributed File System (HDFS) while leveraging the cost-effectiveness and durability of Azure Blob Storage. This article explores the fundamental concepts that underpin ADLS Gen2.

Core Concepts

ADLS Gen2 combines the best of Azure Blob Storage and Azure Data Lake Storage Gen1. Key concepts include:

  • Data Lake: A centralized repository that allows you to store all of your structured and unstructured data at any scale.
  • Hierarchical Namespace: The defining feature of ADLS Gen2, enabling efficient data management and access patterns.
  • Azure Blob Storage Foundation: ADLS Gen2 is an extension of Azure Blob Storage, inheriting its robust features like durability, availability, and cost-effectiveness.
  • POSIX-like ACLs: Provides fine-grained access control for data, essential for multi-user analytical environments.

Hierarchical Namespace

The hierarchical namespace is the most significant differentiator for ADLS Gen2. Unlike the flat namespace of traditional blob storage, ADLS Gen2 organizes data into a hierarchy of directories and files, similar to a file system.

This structure offers several advantages:

  • Optimized performance for analytics: Enables atomic directory operations, making it much faster for big data analytics tasks that frequently involve directory-level operations.
  • Familiar file system model: Simplifies development and management for users accustomed to traditional file systems.
  • Efficient metadata management: The hierarchical structure allows for more efficient organization and retrieval of data.

When you enable the hierarchical namespace on an Azure Storage account, it becomes an ADLS Gen2-enabled account.

Note: You can only enable the hierarchical namespace feature when creating a new storage account. It cannot be enabled or disabled on an existing storage account.

Azure Blob Storage Integration

ADLS Gen2 is built directly on top of Azure Blob Storage. This means that ADLS Gen2 accounts are Azure Storage accounts that have the Hierarchical Namespace feature enabled.

You can use standard Blob Storage APIs and tools to interact with ADLS Gen2 data, alongside the specialized APIs designed for its hierarchical features. This integration allows you to:

  • Use familiar tools like Azure Storage Explorer, Azure CLI, and Azure PowerShell.
  • Leverage Azure Blob Storage features such as lifecycle management, data redundancy options (LRS, GRS, RA-GRS), and hot/cool/archive tiers.
  • Integrate seamlessly with other Azure services that work with Blob Storage.

For example, you can access data in ADLS Gen2 using the Blob Storage endpoint, but with the performance benefits of the hierarchical namespace.

Security and Access Control

ADLS Gen2 provides robust security features, building upon Azure Storage security and adding fine-grained access control.

  • Azure Role-Based Access Control (RBAC): Controls access to the storage account itself and high-level operations.
  • Access Control Lists (ACLs): For data within the ADLS Gen2 file system, POSIX-like ACLs provide granular permissions (read, write, execute) at the directory and file level. These ACLs are inherited and can be modified.
  • Shared Key Authorization: Traditional method for authenticating requests.
  • Service Principal Authorization: Allows applications and services to authenticate securely.
  • Managed Identities: Enables Azure services to authenticate to ADLS Gen2 without managing credentials.

The combination of RBAC and ACLs ensures that you can implement a comprehensive security strategy for your big data analytics data.

Tip: For optimal security, it's recommended to use RBAC for broad access control and ACLs for specific file and directory permissions.

Performance and Scalability

ADLS Gen2 is designed for high-performance big data analytics and massive scalability.

  • Massive Scalability: Can handle exabytes of data and millions of operations per second, scaling automatically to meet demand.
  • Optimized for Analytics: The hierarchical namespace significantly improves the performance of workloads that involve many small files or require frequent directory operations, such as those common in big data processing frameworks like Apache Spark and Hadoop.
  • High Throughput: Designed to deliver high throughput for both read and write operations, crucial for large-scale data processing.
  • Low Latency: Offers low latency access to data, reducing processing times.