Select and Configure Compute in a Workspace

A nightly ETL job and a data scientist exploring a sample in a notebook have opposite needs: the job wants a clean, isolated cluster that appears for the run and disappears afterward, while the notebook user wants a warm cluster they can share and keep alive. Azure Databricks answers both with four distinct compute choices, and choosing the wrong one is the most common configuration mistake on this objective.

Job compute (also called jobs compute) provisions a dedicated cluster scoped to a single job run and terminates it when the run finishes. That isolation keeps scheduled, non-interactive pipelines away from interactive development, and it bills at the lower jobs DBU rate. A DBU (Databricks Unit) is the normalized unit Databricks meters processing by. See Configure compute for jobs^[1].

All-purpose compute is persistent, multi-user compute for notebooks and ad hoc analysis. It stays up so a team can attach to it interactively, and it bills at a higher DBU rate than job compute. Because it mixes development with production and is shared, Databricks does not recommend it for production pipelines. Reaching for all-purpose compute to run a scheduled job is the classic isolation mistake (Classic compute overview^[2]).

A SQL warehouse is compute tuned for SQL queries and BI tools such as Power BI and Tableau, and it comes in serverless, pro, and classic types (SQL warehouse types^[3]). It runs SQL, not notebook or declarative-pipeline code, so it cannot be the compute engine for a Lakeflow Spark Declarative Pipeline (the product formerly branded Delta Live Tables). If a question asks which compute runs a pipeline, a SQL warehouse is always wrong.

Serverless compute runs in the Databricks-managed cloud account and starts in seconds with no VM or instance-pool configuration to manage; it is available for notebooks, jobs, and Lakeflow Spark Declarative Pipelines (Connect to serverless compute^[4]). The trade-off against the other three types is control: classic compute (job, all-purpose, and pro or classic SQL warehouses) runs in your own Azure subscription and exposes every knob, while serverless hides the infrastructure in exchange for instant start and no idle management. Databricks groups the choices into these two families, shown below.

Total cluster compute is simple arithmetic: worker count multiplied by each node's CPU and memory. That gives you two levers to scale a workload, adding more workers or picking a larger node type, and it explains the two special cases below.

A cluster has exactly one driver node plus zero or more worker nodes; the driver coordinates work and the workers run the Spark executors. A single-node cluster sets the worker count to 0, so the driver runs all the Spark work itself. It cannot autoscale and suits only small or non-distributed workloads; a multi-node cluster, by contrast, cannot be scaled down to 0 workers (Compute configuration reference^[5]).

Cluster autoscaling automatically adds and removes worker nodes between a minimum and maximum you set, based on load (autoscaling behavior^[5]). Two facts trip people up. First, autoscaling resizes only workers, never the driver. Second, autoscaling only resizes an active cluster; it never terminates an idle one. Stopping idle compute is the job of automatic termination, covered in the next section, so enabling autoscaling on its own does not remove idle cost.

Streaming workloads are the exception where plain cluster autoscaling scales down poorly. For those, Databricks recommends enhanced autoscaling, the mode Lakeflow Spark Declarative Pipelines use. Enhanced autoscaling allocates cluster resources to workload volume and, unlike cluster autoscaling, proactively shuts down under-utilized nodes while guaranteeing no failed tasks during shutdown (enhanced autoscaling^[6]). It is the default for new pipelines.

Set enhanced autoscaling in a pipeline's cluster configuration

This excerpt of a pipeline's settings turns on enhanced autoscaling; the surrounding pipeline configuration is omitted:

{
  "clusters": [
    {
      "autoscale": { "min_workers": 5, "max_workers": 10, "mode": "ENHANCED" }
    }
  ]
}

The autoscale block bounds the worker pool with min_workers and max_workers (max_workers must be greater than or equal to min_workers), and mode selects ENHANCED over the LEGACY cluster-autoscaling behavior. As the figure below shows, autoscaling of either kind stretches the worker band while the single driver stays put.

Idle compute is where Databricks bills leak, and four independent levers each plug a different leak. Confusing them is a favorite exam trap, so anchor each lever to the specific waste it removes.

Automatic termination shuts a cluster down after a set number of minutes with no Spark jobs, queries, or commands running, removing idle DBU and VM cost (automatic termination^[5]). A cluster counts as inactive only when nothing is executing, so an active workload keeps it alive and is never interrupted. This is the only lever here that stops an idle cluster.

Instance pools hold a set of idle, ready-to-use cloud VMs so that clusters and autoscaling attach pre-acquired nodes and start faster (Connect to pools^[7]). You pay the Azure VM cost for the idle pooled instances, but Databricks charges no DBUs while instances sit idle in the pool. Pools cut start and scale-up latency; they do not by themselves reduce idle-cluster cost.

Azure Spot VMs lower per-hour worker cost by using spare Azure capacity, which Azure can reclaim at any time. The driver is always on-demand; only workers run as spot, and if a spot node is evicted Databricks tries to reacquire capacity and falls back to on-demand (spot instances^[5]). Spot fits fault-tolerant workloads and, like autoscaling, does not stop an idle cluster.

Photon is Databricks' native vectorized query engine, written in C++, that transparently accelerates SQL workloads and Spark DataFrame API calls, including JSON ETL and Delta operations; it is compatible with Apache Spark APIs and needs no code change (What is Photon?^[8]). You turn it on per cluster with the Use Photon Acceleration checkbox, and it is on by default on recent runtimes. One boundary matters for the exam: Photon accelerates SQL and DataFrame operations but does not support user-defined functions (UDFs), RDD, or Dataset APIs, where execution falls back to the Spark runtime.

Keep the reconciling rule at the front of your mind: autoscaling and Spot both lower cost, but neither stops an idle cluster. Only automatic termination does that.

A SQL warehouse scales on a different model than a cluster, and mixing the two vocabularies is a reliable way to miss a question. Three separate settings control a warehouse.

Cluster size is a t-shirt size from X-Small up to 4X-Large (with 2X-Small below it and a 5X-Large in preview) that sets the compute power for a single query; a larger size means a bigger single cluster (SQL warehouse sizing, scaling, and queuing^[9]). Increase the size when queries spill to disk.

Scaling sets a minimum and maximum number of clusters and adds or removes whole clusters to absorb concurrent-query load, not worker nodes (warehouse behavior^[9]). This is the crucial contrast with cluster autoscaling: cluster autoscaling adds worker nodes to one cluster, whereas a SQL warehouse adds entire clusters, and Databricks recommends about one cluster per 10 concurrent queries. The default is a minimum and maximum of one cluster.

Auto Stop shuts an idle warehouse down after a set idle period. It is the warehouse-level equivalent of a cluster's automatic termination and a distinct setting from it (Create a SQL warehouse^[10]). Defaults differ by type: pro and classic warehouses default to 45 minutes (minimum 10), and serverless defaults to 10 minutes (minimum 5 in the UI). Until it stops, an idle warehouse keeps accruing DBU and cloud charges.

The type you pick sets the rest. Serverless warehouses are Databricks-managed and start in a few seconds using Intelligent Workload Management, while pro and classic warehouses run in your Azure subscription and take a few minutes to start (SQL warehouse types^[3]).

Two choices decide what code will even run on a cluster: the Databricks Runtime version and the libraries you add on top of it.

The Databricks Runtime (DBR) is the set of core components on the cluster, and choosing its version fixes the bundled Apache Spark version and the preinstalled packages (Compute configuration reference^[5]). For production, prefer a Long Term Support (LTS) release: LTS versions are supported and patched longer, so operational jobs stay stable and you can test before upgrading (Databricks Runtime release notes^[11]). For data science, the Databricks Runtime for Machine Learning extends the standard runtime with preinstalled ML libraries such as PyTorch, TensorFlow, and scikit-learn, plus GPU variants, so you skip installing the ML stack by hand (Databricks Runtime for Machine Learning^[12]).

Beyond the runtime, libraries come in two scopes that behave differently. Cluster-scoped (compute-scoped) libraries install on the cluster and are available to every notebook and job attached to it. Notebook-scoped libraries, installed with %pip, apply only to the current notebook session, do not affect other notebooks on the same cluster, and do not persist, so they must be reinstalled each session (Install libraries^[13]).

Install a notebook-scoped library

Run %pip inside the notebook to scope a package to that session only:

%pip install databricks-sdk

Because this install is notebook-scoped, the package is gone when the cluster restarts; to make it available to every notebook, install it as a cluster-scoped library instead.

Libraries install from package repositories (PyPI, Maven, CRAN) or from uploaded artifacts such as wheel, egg, or JAR files stored in workspace files or Unity Catalog volumes (Install libraries^[13]). One governance rule interacts with access mode: on standard (formerly shared) access mode compute, libraries and init scripts must be on the Unity Catalog allowlist that a metastore admin maintains, whereas dedicated (single-user) access mode has no such restriction (Allowlist libraries and init scripts^[14]). The allowlist is a standard-access-mode control; it does not apply to dedicated compute.

Four separate controls decide who touches a cluster, and the exam leans on distinguishing them. Lead with the split: creating compute is governed differently than using compute that already exists.

Whether a user can create compute at all depends on an entitlement. A user can create all-purpose compute only if granted the Unrestricted cluster creation entitlement, or a policy-scoped one; without either, they can merely attach to compute they have been given access to (Classic compute overview^[2], Manage entitlements^[15]).

A compute policy (cluster policy) is an admin-defined rule set that limits the compute a user may create or configure: allowed node types and runtime versions, enforced auto-termination, maximum workers or DBU caps, and mandatory tags (Create and manage compute policies^[16]). Granting a user a policy scopes their creation rights to compute that complies with it, which is how admins enforce cost control and standards. Here is the distinction questions probe: a policy governs the compute a user may create (its configuration limits), which is different from a compute ACL that grants use of a cluster that already exists.

Those ACLs assign three permission levels to users, groups, or service principals: CAN ATTACH TO (run commands and view logs), CAN RESTART (start, stop, and restart), and CAN MANAGE (edit configuration and permissions) (Classic compute overview^[2]). Each level includes the ones below it.

Finally, access mode sets sharing and Unity Catalog behavior. Standard access mode supports multiple concurrent users with full Unity Catalog governance, while dedicated access mode is assigned to a single user or group; both are Unity Catalog-enabled (Compute configuration reference^[5]). The legacy No Isolation Shared mode is the exception that is not Unity Catalog-enabled, which is why Databricks steers new work to standard or dedicated (Standard compute requirements and limitations^[17]).

Most questions on this objective hinge on one of a handful of distinctions. Read the stem for the signal, then pick the answer that matches.

• Isolate a scheduled production job: choose job compute (or serverless), not all-purpose compute. All-purpose is shared and interactive and mixes dev with prod.
• Run a Lakeflow Spark Declarative Pipeline: pick job or serverless pipeline compute; a SQL warehouse can never run pipeline or notebook code.
• Stop paying for an idle cluster: the answer is automatic termination. Autoscaling and Spot VMs lower cost but never stop an idle cluster.
• Resize an active cluster to load: cluster autoscaling, which adds and removes workers only, between Min and Max, and never touches the driver.
• Handle many concurrent SQL users: raise a SQL warehouse's maximum clusters (Scaling adds whole clusters), not the t-shirt size, which sets single-query power. Auto Stop, not automatic termination, halts an idle warehouse.
• Speed up cluster start-up: an instance pool of warm VMs. It cuts start time; it is not an idle-cost control.
• Accelerate SQL and DataFrame ETL: enable Photon, but remember it does not cover UDFs, RDD, or Dataset APIs.
• Production stability versus the ML stack: an LTS runtime for operational jobs; the Databricks Runtime for Machine Learning for preinstalled PyTorch, TensorFlow, and scikit-learn.
• Who can spin up compute: the cluster-creation entitlement, plus any compute policy that scopes it. A policy limits what you can create; a compute ACL grants use of an existing cluster.
• Libraries on standard access mode: they must be on the Unity Catalog allowlist; dedicated access mode is not subject to it.