Microsoft Docs

Overview of Azure Data Lake Storage Gen2

Azure Data Lake Storage Gen2 is a highly scalable and secure data lake, built on Azure Blob Storage. It is designed for big data analytics workloads and offers features that are critical for these scenarios, such as hierarchical namespace and high-performance analytics.

What is Azure Data Lake Storage Gen2?

Azure Data Lake Storage Gen2 provides a dedicated analytics experience and file system. It combines the capabilities of Azure Data Lake Storage Gen1 with Azure Blob Storage. This enables you to:

• Analyze all of your data: Store structured, semi-structured, and unstructured data at any scale.
• Optimize for analytics: Leverage the high-performance, hierarchical namespace for efficient big data analytics.
• Secure your data: Implement granular access control and robust security features.
• Integrate with Azure services: Seamlessly integrate with Azure Synapse Analytics, Azure Databricks, HDInsight, and more.

Core Concepts

Azure Data Lake Storage Gen2 is built upon Azure Blob Storage, inheriting its core capabilities while adding specific features for big data analytics:

• Hierarchical Namespace: Unlike traditional blob storage, Data Lake Storage Gen2 organizes data into a hierarchy of directories and files, similar to a file system. This allows for more efficient data management and faster access for analytics engines.
• Blob Storage Compatibility: You can interact with Data Lake Storage Gen2 using the same tools and APIs used for Azure Blob Storage, including REST APIs, Azure SDKs, Azure Storage Explorer, and Azure Data Factory.
• Performance Optimization: The hierarchical namespace significantly improves the performance of analytics operations, especially those that involve listing directories or accessing individual files within a large dataset.
• Security: Data Lake Storage Gen2 supports Azure Role-Based Access Control (RBAC) for managing access at the account and container level, as well as Access Control Lists (ACLs) for fine-grained permissions on directories and files within a data lake.

Use Cases

Data Lake Storage Gen2 is ideal for a wide range of big data and analytics scenarios, including:

• Big data analytics pipelines
• Machine learning and AI workloads
• Data warehousing and business intelligence
• Log analytics and IoT data ingestion
• Real-time streaming analytics