Azure Databricks API Reference

Explore the REST APIs for managing Azure Databricks resources.

Introduction to Databricks REST APIs

The Azure Databricks REST API provides programmatic access to Azure Databricks resources. You can use these APIs to automate tasks such as creating and managing clusters, running jobs, managing notebooks, and interacting with DBFS.

The API endpoints are structured around the Databricks workspace URL. The base URL for the API is:

/api/2.0/

Authentication is typically handled via a Bearer Token, obtained from Azure Active Directory or generated within the Databricks workspace.

Clusters API

Manage Databricks clusters.

Create Cluster

POST /clusters

Creates a new Databricks cluster.

Parameters

Name	Type	Required	Description
cluster_name	String	Yes	The name of the cluster.
spark_version	String	Yes	The Spark version to use.
node_type_id	String	Yes	The VM type for the worker nodes.
num_workers	Integer	No	The number of worker nodes.

Success Response (200 OK)

Field	Type	Description
cluster_id	String	The unique ID of the created cluster.

{
  "cluster_id": "0307-183839-abc123def"
}

Example Request Body

JSON

{
  "cluster_name": "My Spark Cluster",
  "spark_version": "11.3.x-scala2.12",
  "node_type_id": "Standard_DS3_v2",
  "num_workers": 2
}

List Clusters

GET /clusters/list

Lists all clusters in the workspace.

Success Response (200 OK)

Field	Type	Description
clusters	Array of Objects	A list of cluster objects.
clusters[].cluster_id	String	The unique ID of the cluster.
clusters[].cluster_name	String	The name of the cluster.

{
  "clusters": [
    {
      "cluster_id": "0307-183839-abc123def",
      "cluster_name": "My Spark Cluster"
    },
    {
      "cluster_id": "0307-987654-xyz456uvw",
      "cluster_name": "Data Science Cluster"
    }
  ]
}

Jobs API

Manage Databricks jobs.

Create Job

POST /jobs/create

Creates a new Databricks job.

Parameters

Name	Type	Required	Description
name	String	Yes	The name of the job.
new_cluster	Object	No	Configuration for a new cluster to run the job.
existing_cluster_id	String	No	ID of an existing cluster to run the job on.
notebook_task	Object	Yes	Defines the notebook to run.

Example Request Body

JSON

{
  "name": "My Notebook Job",
  "new_cluster": {
    "spark_version": "11.3.x-scala2.12",
    "node_type_id": "Standard_DS3_v2",
    "num_workers": 1
  },
  "notebook_task": {
    "notebook_path": "/Users/user@example.com/MyNotebook"
  }
}

Runs API

Manage job runs.

Run Job

POST /jobs/run-now

Triggers an immediate run of a specified job.

Parameters

Name	Type	Required	Description
job_id	Integer	Yes	The ID of the job to run.

Notebooks API

Manage Databricks notebooks.

Import Notebook

POST /workspace/import

Imports a notebook into the Databricks workspace.

Parameters

Name	Type	Required	Description
path	String	Yes	The destination path for the notebook.
content	String	Yes	The content of the notebook (Base64 encoded if not plain text).
language	String	Yes	The language of the notebook (e.g., 'python', 'scala', 'sql').
overwrite	Boolean	No	If true, overwrite the notebook if it already exists.

Libraries API

Manage libraries installed on clusters.

Install Library

POST /libraries/install

Installs a library on a cluster.

Parameters

Name	Type	Required	Description
cluster_id	String	Yes	The ID of the cluster.
library	Object	Yes	Details of the library to install.

DBFS API

Interact with Databricks File System (DBFS).

List Files

GET /dbfs/list

Lists files and directories in a DBFS path.

Parameters

Name	Type	Required	Description
path	String	Yes	The DBFS path to list.