Microsoft Azure Documentation

Working with Notebooks in Azure Synapse Analytics Spark Pools

Notebooks in Azure Synapse Analytics provide an interactive environment for data professionals to perform exploratory data analysis, data preparation, and advanced analytics using Apache Spark. You can write code in multiple languages (Python, Scala, SQL, .NET) and visualize results directly within the notebook.

Key Features

  • Multi-language Support: Write code in Python (PySpark), Scala, SQL, and .NET.
  • Interactive Execution: Run code cells individually and see results immediately.
  • Data Visualization: Integrate with libraries like Matplotlib, Seaborn, and Plotly for rich data visualizations.
  • Integration with Data Sources: Seamlessly connect to Azure Data Lake Storage, Azure SQL Database, and other data sources.
  • Version Control: Integrate with Git for collaborative development and version management.
  • Managed Spark Environments: Leverage powerful, managed Spark clusters without infrastructure overhead.

Creating and Managing Notebooks

Notebooks are created and managed within the Synapse Studio. You can create new notebooks, open existing ones, and organize them within your workspace.

Steps to Create a Notebook:

  1. Navigate to the Synapse Studio.
  2. In the left pane, select the Develop hub.
  3. Click the + button and select Notebook.
  4. Choose your preferred language and attach it to a Spark pool.

Notebook Code Examples

Here are some basic examples of how to use notebooks:

Python (PySpark) Example: Reading data from Azure Data Lake Storage Gen2

# Sample PySpark code
from pyspark.sql import SparkSession

# Create a SparkSession
spark = SparkSession.builder.appName("ADLSGen2Read").getOrCreate()

# Replace with your actual file path
file_path = "abfss://your-container@your-storage-account.dfs.core.windows.net/path/to/your/data.csv"

# Read data from CSV
df = spark.read.csv(file_path, header=True, inferSchema=True)

# Display the first 5 rows
df.show(5)

# Get schema
df.printSchema()

SQL Example: Querying data using Spark SQL

-- Sample Spark SQL code
SELECT
    column1,
    column2,
    COUNT(*) as record_count
FROM
    `your_database.your_table`
WHERE
    column1 IS NOT NULL
GROUP BY
    column1, column2
ORDER BY
    record_count DESC
LIMIT 10;

API Reference

While Synapse notebooks primarily use Spark APIs, understanding how to interact with Synapse features through code is crucial. Here are some common interactions:

SparkSession Configuration

Method/Property Description
spark.sparkContext.appName Gets the application name of the SparkContext.
spark.conf.set("spark.executor.memory", "4g") Sets a Spark configuration property for executor memory.
spark.conf.get("spark.driver.host") Retrieves the Spark driver host configuration.

Dataframe Operations

Method Description
df.select() Selects a set of columns from a DataFrame.
df.filter() Filters rows of a DataFrame based on a condition.
df.groupBy() Groups the rows using the specified columns.
df.write.saveAsTable() Saves a DataFrame as a managed table in the Spark catalog.

Best Practices

  • Optimize Data Reads: Use appropriate file formats (e.g., Parquet, Delta Lake) and partition your data effectively.
  • Manage Dependencies: Install necessary libraries using the library management features in Synapse Studio.
  • Monitor Performance: Regularly check Spark UI and Synapse monitoring tools to identify performance bottlenecks.
  • Use Git Integration: Commit your notebooks regularly to a Git repository for collaboration and backup.
  • Efficient Spark Queries: Write optimized Spark SQL or DataFrame operations to minimize resource usage.