> ## Documentation Index
> Fetch the complete documentation index at: https://docs.maia.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Databricks Jobs Compute configuration

export const maia = "Maia";

Run and schedule {maia} transformation [pipelines](/docs/guides/designer-overview#designer-pipelines) on a [Databricks Jobs Compute cluster](https://docs.databricks.com/aws/en/jobs/compute). To enable this, you *must* [create](#create-jobs-compute-configurations) a Jobs Compute configuration in {maia} and [associate](#associating-jobs-compute-configurations-with-an-environment) it with an environment using the Cloud Platform Databricks integration. This configuration provides the necessary details for Databricks to launch the Jobs Compute cluster where your transformation pipeline will execute.

***

## Video example

<iframe width="560" height="315" src="https://www.youtube.com/embed/EMit_-Q2U5w?si=9tdhaiuWrBV0paB0&enablejsapi=1" title="YouTube video player" frameBorder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share; fullscreen" referrerPolicy="strict-origin-when-cross-origin" allowFullScreen />

***

## Why use Jobs Compute?

Databricks Jobs Compute clusters offer a range of benefits tailored for cost-effective data transformations.

### Automated and optimized resource usage

* **Ephemeral clusters:** Jobs Compute clusters are typically spun up on demand and automatically terminated when the job completes. This ensures resources are only used when needed, reducing idle time.
* **Cost efficiency:** Since you only pay for compute while the job is running, this model can significantly lower costs compared to using always-on interactive clusters.

### Environment consistency

* **Repeatable execution:** Each job starts in a fresh cluster environment, which helps avoid problems caused by lingering dependencies or residual data from previous runs.
* **Controlled configurations:** You define the Databricks Runtime version, node type, and number of workers, ensuring consistency across development, staging, and production runs.

***

## When to use Jobs Compute

Jobs Compute is ideal for transformation pipelines that don't require low-latency or real-time execution. Since these clusters are created on demand, there's a much longer startup time compared to an all purpose cluster. This makes Jobs Compute a cost-effective choice for scheduled or batch workloads, where reduced infrastructure costs are prioritized over immediate responsiveness. It offers a reliable option for teams focused on optimizing resource usage.

***

## Accessing Jobs Compute

The Jobs Compute feature is available at the [project](/docs/guides/projects) level. To access it:

1. In {maia}, go to the **Your projects** menu, and select a Databricks project.
2. Along the top, you will see a list of resource tabs. Click **More** on the far right, then select **Job Compute**. A list of your existing Jobs Compute configurations will be displayed.

This is where you can manage your Jobs Compute configurations. In this dialog you can:

* [Create Jobs Compute configurations](#create-jobs-compute-configurations).
* [Edit Jobs Compute details](#edit-jobs-compute-details).
* [Delete Jobs Compute configurations](#delete-jobs-compute-configurations).

### Create Jobs Compute configurations

To create Jobs Compute configurations that can later be [associated](#associating-jobs-compute-configurations-with-an-environment) with your environment, follow these steps:

1. From the Jobs Compute configurations page, click the **Add jobs compute** button at the top.
2. The **Create a jobs compute Configuration** dialog will appear. Complete the following fields:
   * **Name:** Enter a unique name to identify the configuration.
   * **Policy ID**: Enter the ID of a single Databricks compute policy. Read [Create and manage compute policies](https://docs.databricks.com/aws/en/admin/clusters/policies) to learn more.
   * **Access mode:** Use the drop-down menu to select the access mode to be used by the cluster. For example, **No isolation shared**, **Shared**, and **Single User**. For more information, read [Access modes](https://docs.databricks.com/aws/en/compute/configure#access-modes).
   * **Worker type:** Use the drop-down menu to select the size of the node type to be used by the cluster. For example, **r6id.xlarge**, **i3.xlarge**.
   * **Databricks runtime version:** Use the drop-down menu to select the runtime version that will be used by the cluster. For example, **15.4LTS**. For more information, read [Databricks Runtime release notes versions and compatibility](https://docs.databricks.com/aws/en/release-notes/runtime).
   * **Number of workers:** Enter the number of worker nodes you want Databricks to allocate for running the job. Enter a number between **0** and **100,000**.
3. Click **Create** to save the configuration.

You will return to the Jobs Compute configurations page.

### Edit Jobs Compute details

To edit an existing Jobs Compute configuration, following these steps:

1. On the Jobs Compute configurations page, click the three dots **...** next to the configuration you want to modify.
2. Click **Edit Configuration**.
3. The **Edit a jobs compute configuration** page will be open, allowing you to update the details specified during the initial setup. For more information, read [Create a Jobs Compute configuration](#create-jobs-compute-configurations).
4. Click **Update** to save your changes.

You will return to the Jobs Compute configurations page.

### Delete Jobs Compute configurations

To delete an existing Jobs Compute configuration, follow these steps:

1. On the Jobs Compute configurations page, click the three dots **...** next to the configuration you want to delete.
2. Click **Delete Configuration**.
3. A confirmation dialog will appear. Click **Yes, delete** to proceed.

   <Note>
     This action can't be undone, and will impact any environment or transformation pipeline where the Jobs Compute configuration is in use.
   </Note>

Once confirmed, the configuration will be removed from your list.

***

## Associating Jobs Compute configurations with an environment

A Jobs Compute configuration *must* be created on the Jobs Compute configurations page before it can be associated with an environment. For more information, read [Create a Jobs Compute configuration](#create-jobs-compute-configurations).

To associate a Jobs Compute configuration with an environment, follow these steps:

1. In {maia}, go to the **Your projects** menu.
2. Along the top you will see a list of resource tabs. Click **Environments**.
3. A list of your environments will be displayed. For more information, read [Environments](/docs/guides/environments). Click the three dots **...** next to the environment you want to update.
4. Click **Associate jobs compute**.
5. The **Environment name** field will be pre-populated with your default environment. Use the drop-down menu to select the Jobs Compute configuration you want to associate.
6. Click **Associate**.

All transformation pipelines in this environment will now run using the selected Jobs Compute configuration.