Select and Configure Compute in a Workspace

Match the compute to the workload

A nightly ETL job and a data scientist exploring a sample in a notebook have opposite needs: the job wants a clean, isolated cluster that appears for the run and disappears afterward, while the notebook user wants a warm cluster they can share and keep alive. Azure Databricks answers both with four distinct compute choices, and choosing the wrong one is the most common configuration mistake on this objective.

Job compute (also called jobs compute) provisions a dedicated cluster scoped to a single job run and terminates it when the run finishes. That isolation keeps scheduled, non-interactive pipelines away from interactive development, and it bills at the lower jobs DBU rate. A DBU (Databricks Unit) is the normalized unit Databricks meters processing by. See Configure compute for jobs^[1].

All-purpose compute is persistent, multi-user compute for notebooks and ad hoc analysis. It stays up so a team can attach to it interactively, and it bills at a higher DBU rate than job compute. Because it mixes development with production and is shared, Databricks does not recommend it for production pipelines. Reaching for all-purpose compute to run a scheduled job is the classic isolation mistake (Classic compute overview^[2]).

A SQL warehouse is compute tuned for SQL queries and BI tools such as Power BI and Tableau, and it comes in serverless, pro, and classic types (SQL warehouse types^[3]). It runs SQL, not notebook or declarative-pipeline code, so it cannot be the compute engine for a Lakeflow Spark Declarative Pipeline (the product formerly branded Delta Live Tables). If a question asks which compute runs a pipeline, a SQL warehouse is always wrong.

Serverless compute runs in the Databricks-managed cloud account and starts in seconds with no VM or instance-pool configuration to manage; it is available for notebooks, jobs, and Lakeflow Spark Declarative Pipelines (Connect to serverless compute^[4]). Databricks groups the choices into two families, and the figure below draws them: classic compute, holding all-purpose compute, job compute, and pro or classic SQL warehouses, and Databricks-managed serverless compute, holding notebooks, jobs, pipelines, and the serverless SQL warehouse. The trade-off against the other three types is control: classic compute (job, all-purpose, and pro or classic SQL warehouses) runs in your own Azure subscription and exposes every knob, while serverless hides the infrastructure in exchange for instant start and no idle management.

Two compute families: classic compute runs in your Azure subscription; serverless compute is Databricks-managed. Source: Azure Databricks compute docs.

Size a cluster: workers, nodes, and autoscaling

Total cluster compute is simple arithmetic: worker count multiplied by each node's CPU and memory. That gives you two levers to scale a workload, adding more workers or picking a larger node type, and it explains the two special cases below.

A cluster has exactly one driver node plus zero or more worker nodes; the driver coordinates work and the workers run the Spark executors. A single-node cluster sets the worker count to 0, so the driver runs all the Spark work itself. It cannot autoscale and suits only small or non-distributed workloads; a multi-node cluster, by contrast, cannot be scaled down to 0 workers (Compute configuration reference^[5]).

Cluster autoscaling automatically adds and removes worker nodes between a minimum and maximum you set, based on load (autoscaling behavior^[5]). Two facts trip people up. First, autoscaling resizes only workers, never the driver. Second, autoscaling only resizes an active cluster; it never terminates an idle one. Stopping idle compute is the job of automatic termination, covered in the next section, so enabling autoscaling on its own does not remove idle cost.

Streaming workloads are the exception where plain cluster autoscaling scales down poorly. For those, Databricks recommends enhanced autoscaling, the mode Lakeflow Spark Declarative Pipelines use. Enhanced autoscaling allocates cluster resources to workload volume and, unlike cluster autoscaling, proactively shuts down under-utilized nodes while guaranteeing no failed tasks during shutdown (enhanced autoscaling^[6]). It is the default for new pipelines. As the figure below shows, autoscaling of either kind stretches the worker band while the single driver stays put.

Set enhanced autoscaling in a pipeline's cluster configuration

This excerpt of a pipeline's settings turns on enhanced autoscaling; the surrounding pipeline configuration is omitted:

{
  "clusters": [
    {
      "autoscale": { "min_workers": 5, "max_workers": 10, "mode": "ENHANCED" }
    }
  ]
}

The autoscale block bounds the worker pool with min_workers and max_workers (max_workers must be greater than or equal to min_workers), and mode selects ENHANCED over the LEGACY cluster-autoscaling behavior.

One driver node stays fixed while autoscaling adds and removes worker nodes between Min and Max. Source: Azure Databricks compute configuration reference.

Spend less without losing the work

Idle compute is where Databricks bills leak, and each cost lever plugs a different leak. Cluster autoscaling from the previous section is one of them; four more are set per cluster, and this section takes those four. Confusing them is a favorite exam trap, so anchor each lever to the specific waste it removes.

Automatic termination shuts a cluster down after a set number of minutes with no Spark jobs, queries, or commands running, removing idle DBU and VM cost (automatic termination^[5]). A cluster counts as inactive only when nothing is executing, so an active workload keeps it alive and is never interrupted. This is the only lever here that stops an idle cluster.

Instance pools hold a set of idle, ready-to-use cloud VMs so that clusters and autoscaling attach pre-acquired nodes and start faster (Connect to pools^[7]). You pay the Azure VM cost for the idle pooled instances, but Databricks charges no DBUs while instances sit idle in the pool. Pools cut start and scale-up latency; they do not by themselves reduce idle-cluster cost.

Azure Spot VMs lower per-hour worker cost by using spare Azure capacity, which Azure can reclaim at any time. The driver is always on-demand; only workers run as spot, and if a spot node is evicted Databricks tries to reacquire capacity and falls back to on-demand (spot instances^[5]). Spot fits fault-tolerant workloads and, like autoscaling, does not stop an idle cluster.

Photon is Databricks' native vectorized query engine, written in C++, that transparently accelerates SQL workloads and Spark DataFrame API calls, including JSON ETL and Delta operations; it is compatible with Apache Spark APIs and needs no code change (What is Photon?^[8]). You turn it on per cluster with the Use Photon Acceleration checkbox, and it is on by default on recent runtimes. One boundary matters for the exam: Photon accelerates SQL and DataFrame operations but does not support user-defined functions (UDFs), RDD, or Dataset APIs, where execution falls back to the Spark runtime.

The figure below pairs each cost problem with the one lever that removes it, cluster autoscaling included. Keep the reconciling rule at the front of your mind: autoscaling and Spot both lower cost, but neither stops an idle cluster. Only automatic termination does that.

Each lever removes a different waste; only automatic termination stops an idle cluster. Source: Azure Databricks compute configuration reference.

SQL warehouses: size, scale, and Auto Stop

A SQL warehouse scales on a different model than a cluster, and mixing the two vocabularies is a reliable way to miss a question. Three separate settings control a warehouse.

Cluster size is a t-shirt size from X-Small up to 4X-Large (with 2X-Small below it and a 5X-Large in preview) that sets the compute power for a single query; a larger size means a bigger single cluster (SQL warehouse sizing, scaling, and queuing^[9]). Increase the size when queries spill to disk.

Scaling sets a minimum and maximum number of clusters and adds or removes whole clusters to absorb concurrent-query load, not worker nodes (warehouse behavior^[9]). This is the crucial contrast with cluster autoscaling: cluster autoscaling adds worker nodes to one cluster, whereas a SQL warehouse adds entire clusters, and Databricks recommends about one cluster per 10 concurrent queries. The default is a minimum and maximum of one cluster.

Auto Stop shuts an idle warehouse down after a set idle period. It is the warehouse-level equivalent of a cluster's automatic termination and a distinct setting from it (Create a SQL warehouse^[10]). Defaults differ by type: pro and classic warehouses default to 45 minutes (minimum 10), and serverless defaults to 10 minutes (minimum 5 in the UI). Until it stops, an idle warehouse keeps accruing DBU and cloud charges.

The type you pick sets the rest. Serverless warehouses are Databricks-managed and start in a few seconds using Intelligent Workload Management, while pro and classic warehouses run in your Azure subscription and take a few minutes to start (SQL warehouse types^[3]). Hold the three settings together: cluster size sets how much power one query gets, Scaling adds whole clusters to absorb concurrent queries where a cluster's autoscaling adds worker nodes, and Auto Stop, not automatic termination, is what stops an idle warehouse.

Choose a Databricks Runtime version

Two choices decide what code will even run on a cluster: the Databricks Runtime version and the libraries you add on top of it. Start with the runtime; the libraries follow in the next section.

The Databricks Runtime (DBR) is the set of core components on the cluster, and choosing its version fixes the bundled Apache Spark version and the preinstalled packages (Compute configuration reference^[5]). For production, prefer a Long Term Support (LTS) release: LTS versions are supported and patched longer, so operational jobs stay stable and you can test before upgrading (Databricks Runtime release notes^[11]). For data science, the Databricks Runtime for Machine Learning extends the standard runtime with preinstalled ML libraries such as PyTorch, TensorFlow, and scikit-learn, plus GPU variants, so you skip installing the ML stack by hand (Databricks Runtime for Machine Learning^[12]).

The runtime choice is a stability-versus-stack trade: take an LTS release for anything operational, and the Databricks Runtime for Machine Learning when you want the data-science stack already installed.

Manage cluster-scoped and notebook-scoped libraries

Beyond the runtime, libraries come in two scopes that behave differently. Cluster-scoped (compute-scoped) libraries install on the cluster and are available to every notebook and job attached to it. Notebook-scoped libraries, installed with %pip, apply only to the current notebook session, do not affect other notebooks on the same cluster, and do not persist, so they must be reinstalled each session (Install libraries^[13]).

Install a notebook-scoped library

Run %pip inside the notebook to scope a package to that session only:

%pip install databricks-sdk

Because this install is notebook-scoped, the package is gone when the cluster restarts; to make it available to every notebook, install it as a cluster-scoped library instead.

Libraries install from package repositories (PyPI, Maven, CRAN) or from uploaded artifacts such as wheel, egg, or JAR files stored in workspace files or Unity Catalog volumes (Install libraries^[13]). One governance rule interacts with access mode: on standard (formerly shared) access mode compute, libraries and init scripts must be on the Unity Catalog allowlist that a metastore admin maintains, whereas dedicated (single-user) access mode has no such restriction (Allowlist libraries and init scripts^[14]). The allowlist is a standard-access-mode control; it does not apply to dedicated compute.

Govern who can create and attach compute

Four separate controls decide who touches a cluster, and the exam leans on distinguishing them. Lead with the split: creating compute is governed differently than using compute that already exists.

Whether a user can create compute at all depends on an entitlement. A user can create all-purpose compute only if granted the Unrestricted cluster creation entitlement, or a policy-scoped one; without either, they can merely attach to compute they have been given access to (Classic compute overview^[2], Manage entitlements^[15]).

A compute policy (cluster policy) is an admin-defined rule set that limits the compute a user may create or configure: allowed node types and runtime versions, enforced auto-termination, maximum workers or DBU caps, and mandatory tags (Create and manage compute policies^[16]). Granting a user a policy scopes their creation rights to compute that complies with it, which is how admins enforce cost control and standards. Here is the distinction questions probe: a policy governs the compute a user may create (its configuration limits), which is different from a compute ACL that grants use of a cluster that already exists.

Those ACLs assign three permission levels to users, groups, or service principals: CAN ATTACH TO (run commands and view logs), CAN RESTART (start, stop, and restart), and CAN MANAGE (edit configuration and permissions) (Classic compute overview^[2]). Each level includes the ones below it.

Finally, access mode sets sharing and Unity Catalog behavior. Standard access mode supports multiple concurrent users with full Unity Catalog governance, while dedicated access mode is assigned to a single user or group; both are Unity Catalog-enabled (Compute configuration reference^[5]). The legacy No Isolation Shared mode is the exception that is not Unity Catalog-enabled, which is why Databricks steers new work to standard or dedicated (Standard compute requirements and limitations^[17]).

Exam-pattern recognition

Most questions on this objective hinge on one of a handful of distinctions. Read the stem for the signal, then pick the answer that matches.

Isolate a scheduled production job: choose job compute (or serverless), not all-purpose compute. All-purpose is shared and interactive and mixes dev with prod.
Run a Lakeflow Spark Declarative Pipeline: pick job or serverless pipeline compute; a SQL warehouse can never run pipeline or notebook code.
Stop paying for an idle cluster: the answer is automatic termination. Autoscaling and Spot VMs lower cost but never stop an idle cluster.
Resize an active cluster to load: cluster autoscaling, which adds and removes workers only, between Min and Max, and never touches the driver.
Handle many concurrent SQL users: raise a SQL warehouse's maximum clusters (Scaling adds whole clusters), not the t-shirt size, which sets single-query power. Auto Stop, not automatic termination, halts an idle warehouse.
Speed up cluster start-up: an instance pool of warm VMs. It cuts start time; it is not an idle-cost control.
Accelerate SQL and DataFrame ETL: enable Photon, but remember it does not cover UDFs, RDD, or Dataset APIs.
Production stability versus the ML stack: an LTS runtime for operational jobs; the Databricks Runtime for Machine Learning for preinstalled PyTorch, TensorFlow, and scikit-learn.
Who can spin up compute: the cluster-creation entitlement, plus any compute policy that scopes it. A policy limits what you can create; a compute ACL grants use of an existing cluster.
Libraries on standard access mode: they must be on the Unity Catalog allowlist; dedicated access mode is not subject to it.

Choosing among job, all-purpose, SQL warehouse, and serverless compute

Consideration	Job compute	All-purpose compute	SQL warehouse	Serverless compute
Primary use	Scheduled production runs	Interactive notebooks and ad hoc	SQL queries and BI tools	On-demand notebooks, jobs, pipelines
Lifecycle	Created per run, then terminated	Persistent, multi-user	Started and stopped by Auto Stop	Databricks-managed, starts in seconds
Sharing and isolation	Isolated to one run	Shared across users	Shared query engine	Managed and isolated by Databricks
Relative DBU rate	Lower jobs rate	Higher all-purpose rate	SQL warehouse rate	Serverless rate
Runs pipeline or notebook code?	Yes	Yes	No (SQL and BI only)	Yes

Decision tree

Cheat sheet

Sharp facts the exam loves — give these one last read before exam day.

Cheat sheet

Sharp facts the exam loves — scan these before test day.

Job compute is dedicated, ephemeral compute for scheduled production runs

Job (classic jobs) compute provisions a dedicated cluster scoped to a single job run and terminates it when the run finishes, isolating scheduled non-interactive pipelines from interactive development and billing at the lower jobs DBU rate.

Trap All-purpose compute is interactive and shared, so it is not recommended for production jobs and breaks the isolation requirement.

10 questions test this

All-purpose (shared) compute is for interactive, collaborative development

All-purpose compute is persistent, multi-user interactive compute for notebooks and ad hoc analysis; it bills at a higher DBU rate than job compute and is not recommended for production pipelines because it mixes development and production workloads.

10 questions test this

SQL warehouses run SQL and BI workloads, not pipeline code

A SQL warehouse is compute optimized for SQL queries and BI tools such as Power BI and Tableau, and comes in serverless, pro, and classic types; it cannot execute notebook or declarative-pipeline code.

Trap A SQL warehouse cannot serve as the compute engine for a Lakeflow declarative pipeline.

11 questions test this

Serverless compute starts instantly and is managed by Databricks

Serverless compute runs in the Databricks-managed cloud account and starts in seconds with no VM configuration or instance-pool management; it is available for notebooks, jobs, and declarative pipelines.

7 questions test this

Autoscaling adds and removes only worker nodes between a min and max

Cluster autoscaling automatically adds or removes worker nodes between a configured minimum and maximum based on load; it resizes only workers, never the driver, and it does not stop an idle cluster.

Trap Autoscaling only resizes an active cluster; it never terminates an idle one, which is the job of automatic termination.

13 questions test this

A single-node cluster has zero workers and runs on the driver only

A single-node cluster sets the worker count to 0 so the driver runs all Spark work with no worker nodes; it cannot autoscale and suits only small, non-distributed workloads.

4 questions test this

Enhanced autoscaling optimizes Lakeflow declarative-pipeline compute

Lakeflow Spark Declarative Pipelines run on runtime-managed job compute that uses enhanced autoscaling, which scales workers to workload volume and proactively shuts down under-utilized nodes to minimize cost.

3 questions test this

Cluster capacity is worker count multiplied by per-node CPU and memory

Total cluster compute equals the number of worker nodes multiplied by each node's CPU and memory, so a workload is scaled either by adding workers (node count) or by selecting a larger node type.

SQL warehouse size sets query power; scaling adds clusters; Auto Stop halts idle

A SQL warehouse t-shirt cluster size (X-Small to 4X-Large) sets the compute power for a single query, while its Scaling min/max range adds or removes WHOLE clusters to absorb concurrent-query load (not worker nodes, unlike cluster autoscaling); Auto Stop shuts an idle warehouse down (default 45 minutes classic/pro, 10 minutes serverless).

Trap SQL warehouse Scaling adds whole CLUSTERS for concurrency, whereas cluster autoscaling adds WORKER NODES; Auto Stop (warehouse) is distinct from a cluster automatic termination.

5 questions test this

Automatic termination shuts down a cluster after idle minutes

Automatic termination stops an all-purpose cluster after a set number of minutes with no Spark jobs, queries, or commands running, removing idle DBU and VM cost; a cluster is inactive only when nothing is executing, so active workloads keep it alive and are never interrupted.

Trap Enabling autoscaling or spot pricing lowers cost but does not stop an idle cluster; only automatic termination removes the idle waste.

13 questions test this

Instance pools keep warm VMs to reduce cluster start and scale time

An instance pool holds a set of idle, ready-to-use cloud VMs so clusters and autoscaling attach pre-acquired nodes and start faster; you pay the Azure VM cost for idle pooled instances but no DBU until a cluster uses them.

13 questions test this

Spot VMs reduce worker cost but can be reclaimed

Configuring workers as Azure Spot VMs lowers per-hour compute cost, but Azure can reclaim spot capacity at any time, so spot fits fault-tolerant workloads and does not by itself reduce idle-cluster waste.

Photon is the vectorized engine that accelerates SQL and DataFrame work

Photon is Databricks' native vectorized C++ query engine that transparently accelerates SQL and Spark DataFrame workloads, including JSON ETL and Delta operations; it is turned on per cluster with the Use Photon Acceleration checkbox and is on by default on recent runtimes.

Trap Photon accelerates SQL and DataFrame operations, not arbitrary Python or Scala UDFs.

6 questions test this

The Databricks Runtime version fixes the bundled Spark version

Choosing a Databricks Runtime (DBR) version sets the bundled Apache Spark version and preinstalled libraries; Long Term Support (LTS) releases are recommended for production because they are supported and patched longer.

6 questions test this

Databricks Runtime for Machine Learning preinstalls ML libraries

The Databricks Runtime for Machine Learning extends the standard runtime with preinstalled ML libraries such as PyTorch, TensorFlow, and scikit-learn (with GPU variants), so selecting it avoids manually installing the ML stack.

5 questions test this

Cluster-scoped libraries load for every notebook; notebook-scoped are session only

Cluster-scoped libraries install on the cluster and are available to every notebook attached to it, whereas notebook-scoped libraries installed with %pip apply only to the current notebook session and are discarded when the cluster restarts.

9 questions test this

Libraries install from package repositories or from files and volumes

Compute libraries can be installed from package repositories (PyPI, Maven, CRAN) or from uploaded artifacts such as wheel, egg, or JAR files stored in workspace files or Unity Catalog volumes.

8 questions test this

Standard access mode installs only allowlisted libraries

On standard (formerly shared) access mode compute, libraries and init scripts must be on the Unity Catalog allowlist maintained by a metastore admin, whereas dedicated access mode has no such restriction.

Trap The Unity Catalog allowlist governs standard access mode; dedicated (single-user) compute is not subject to it.

5 questions test this

Compute ACLs grant CAN ATTACH TO, CAN RESTART, or CAN MANAGE

Compute access control assigns three permission levels to users, groups, or service principals: CAN ATTACH TO (run commands and view logs), CAN RESTART (start, stop, and restart), and CAN MANAGE (edit configuration and permissions).

14 questions test this

Access mode (standard vs dedicated) sets sharing and Unity Catalog support

A cluster's access mode determines sharing: standard access mode supports multiple concurrent users with full Unity Catalog governance, while dedicated access mode is assigned to a single user or group; both are Unity Catalog-enabled.

Trap The legacy No Isolation Shared access mode is not Unity Catalog-enabled.

6 questions test this

Creating compute requires the cluster-creation entitlement

A user can create all-purpose compute only if granted the unrestricted or policy-scoped cluster-creation entitlement; without it they can merely attach to existing compute they have been given access to.

Cluster policies constrain what compute a user can create

A cluster (compute) policy is an admin-defined rule set that limits the compute a user may create or configure - allowed node types and Databricks Runtime versions, enforced auto-termination, maximum workers or DBU caps, and mandatory tags. Granting a user a policy scopes their cluster-creation entitlement to compute that complies with it, enforcing cost control and standards.

Trap A cluster policy governs the compute a user may CREATE (its configuration limits); that is different from a cluster ACL that grants Can Attach/Can Manage on an already-existing cluster.

15 questions test this

The Azure Databricks disk cache stores copies of remote Parquet files as local files on the worker's SSD, so a repeatedly re-read data set is served by choosing a worker type with local SSD volumes rather than by adding memory

Disk caching copies remote Parquet data files (including Delta Lake tables) into the worker node's local storage on the first read, automatically and with no code change, and later reads of the same files are served locally. The cache is stored as local files on the worker node and is enabled and configured by default on worker types that ship with SSD volumes, using a bounded share of that local SSD space. This is a different mechanism from the Apache Spark cache, which holds in-memory blocks, must be triggered manually in code, and is not what an analytical or ML workload that repeatedly rescans the same tables relies on; in SQL warehouses and on current Databricks Runtimes the CACHE SELECT command is ignored in favor of the automatic disk cache.

Trap Believing the disk cache is an in-memory cache, and therefore selecting a memory-optimized worker type (or running CACHE SELECT / .persist() to prime it) so that a repeatedly re-read data set is cached - when the disk cache is written to the worker's local SSD, is populated automatically on first read, and is only enabled by default on SSD-backed worker types.

The driver node type defaults to the worker node type and must be enlarged separately for driver-side load, because neither more workers nor a higher autoscaling maximum adds driver capacity

In classic compute the driver runs the Spark master, interprets every command, and holds the state of all notebooks attached to the resource, so its instance type is a separate decision from the worker type even though it defaults to the same value. Databricks tells you to pick a larger driver type with more memory when the workload collects a lot of data back from the executors for analysis in the notebook, or when many notebooks stay attached. Autoscaling min and max settings govern worker nodes only, so driver-side pressure is resolved by changing the driver type (or detaching unused notebooks), not by resizing the worker fleet.

Trap Treating a driver out-of-memory failure during a large collect() as a cluster-capacity problem and fixing it by adding workers or raising the autoscaling maximum - the collected result lands on the driver, and worker count and autoscaling bounds never change the driver's instance type or memory.

Classic and pro SQL warehouses admit a fixed number of concurrent queries per cluster, a ratio a larger cluster size does not raise

For classic and pro SQL warehouses the concurrency model fixes how many concurrent queries one cluster admits, so a single cluster takes at most that many at once and everything beyond it waits in a bounded queue. Raising the t-shirt cluster size buys a bigger driver and more workers for each individual query but leaves that per-cluster admission limit untouched, so serving more simultaneous users means raising the maximum number of clusters and letting the warehouse scale out horizontally. A serverless warehouse instead has its workload manager add and release clusters itself as queue wait moves, within the maximum you configure.

Trap Assuming per-cluster query concurrency scales with the cluster size - expecting a bigger t-shirt size to run more simultaneous queries - and answering a growing-user-count problem by sizing up instead of raising the maximum cluster count.

Bytes spilled to disk in the query profile calls for a larger warehouse cluster size, while a persistently non-zero Peak Queued Queries metric calls for more clusters

The two right-sizing signals for a SQL warehouse point at two different settings. Spill reported as Bytes spilled to disk in a query's profile means one query exceeded the memory of the cluster it ran on, and the documented remedy is to increase the cluster size. Peak Queued Queries staying above zero on the warehouse monitoring page means demand exceeds admitted capacity, and the remedy for concurrent load is a higher maximum cluster count; for serverless warehouses Intelligent Workload Management then provisions those extra clusters automatically as queue wait grows.

Trap Reading spill to disk as a capacity shortage and raising the maximum cluster count to fix it - extra clusters only add parallel query slots and give the spilling query no additional memory, because a single query still executes on one cluster.

A shuffle- and memory-heavy transformation that spills is addressed by moving the workers to a memory-optimized instance family, not by adding more workers of the same type.

Wide transformations such as large joins and aggregations hold shuffle data and cached DataFrames in executor memory, and when a node's memory-to-core ratio is too low the stage spills and slows. A memory-optimized worker family raises the memory available per core so those partitions fit, whereas adding nodes of the same type raises total capacity while leaving the per-executor ratio, and therefore the spilling, unchanged.

Trap That a spilling job is fixed by raising the worker count or the autoscaling maximum, on the reasoning that more nodes means more total memory.

2 questions test this

A CPU-bound workload with little shuffle or caching belongs on a compute-optimized worker family, which supplies the cores it needs without paying for memory it will not use.

Work dominated by decoding, parsing, or per-record computation, including many streaming jobs that keep only small state, is limited by cores rather than by memory. A compute-optimized family delivers the most CPU per unit of cost for that profile, while a memory-optimized family chosen for the same core count buys memory the job never touches.

Trap That a memory-optimized family is the safe default for every Spark workload because Spark is an in-memory engine.

1 question tests this

You have an Azure Databricks workspace that contains hundreds of Unity Catalog managed Delta tables. A nightly Lakeflow job runs OPTIMIZE and VACUUM across those tables on classic jobs compute. The ma

A driver serving many attached notebooks or collecting results back to itself is selected for driver-side memory, so a shared interactive cluster can need a memory-rich driver family while its workers stay on a cheaper one.

All-purpose compute hosts one interpreter and its session state per attached notebook on the driver, and collected results and broadcast tables are assembled there as well, so a cluster shared by many analysts is constrained on the driver long before the workers are. The lever is the driver's instance family, chosen independently of the workers': extra cores on a compute-optimized driver do not hold more state, and a job cluster running a single automated task rarely needs the same treatment.

Trap That the driver should simply mirror whatever family the workers use, so a cluster shared by many notebooks is tuned by changing the worker type instead.

1 question tests this

You have an Azure Databricks workspace with a shared all-purpose compute resource that about 30 analysts use for interactive notebooks. During busy periods the notebooks become unresponsive and the dr

Serverless notebook and job compute is configured through its environment, an environment version plus declared dependencies, rather than by selecting a Databricks Runtime, attaching a cluster-scoped library, or running an init script.

A serverless workload exposes no cluster specification to edit, so its client image and library baseline come from the serverless environment version that is selected, and extra packages are declared as dependencies of that environment. Where many notebooks and jobs must share one dependency set, a workspace base environment defines it once and is referenced by each workload rather than being installed again per notebook.

Trap That libraries and engine version are pinned on serverless the same way as on classic compute, by choosing a runtime version and attaching cluster-scoped libraries.

2 questions test this

Serverless performance mode is a latency-for-cost lever: performance-optimized buys faster start-up and execution at a higher rate, while standard performance is the cheaper setting for latency-tolerant work.

Both modes run the same code and differ in how aggressively capacity is provisioned for the workload. Interactive notebooks and short user-facing jobs are the case for performance-optimized, while scheduled batch work that nobody is waiting on belongs in standard performance mode so the saving is taken where the delay costs nothing.

Trap That standard performance mode is a smaller machine that risks failing the job, or that performance mode governs how many concurrent workloads are admitted rather than latency and price.

1 question tests this

You have an Azure Databricks workspace that runs a nightly Lakeflow job on serverless compute. The job aggregates the previous day's telemetry, and its results are not read until the following morning

A stated requirement that serverless does not support, such as a custom init script, a specific node type, spot capacity, or an instance pool, is answered by moving the workload to classic compute rather than by tuning serverless.

Serverless hides the cluster, so every setting that names infrastructure is unavailable by design and cannot be reintroduced through the environment. When a scenario fixes such a requirement the correct answer is classic compute, job compute for scheduled runs and all-purpose for interactive work; when it fixes only outcomes such as immediate start, no idle capacity, and no cluster administration, serverless is the answer.

Trap That any serverless limitation can be worked around inside serverless, for example by supplying an init script or requesting a node type through the environment.

2 questions test this

Warehouse type is the lever for start-up latency and management model: a serverless warehouse becomes available in seconds on Databricks-managed capacity, while pro and classic warehouses run in the customer's own cloud account and take minutes to start.

A dashboard that must not wait on a cold warehouse, or a bursty pattern where capacity should disappear between peaks, is answered by choosing the serverless type, not by enlarging a pro warehouse. Pro is the type to keep when the compute must remain inside the customer's cloud account, and classic is the entry-level type that lacks the newer performance capabilities.

Trap That cold-start delay on a pro or classic warehouse is removed by a larger cluster size or a longer auto-stop timeout.

2 questions test this

Intelligent Workload Management is a serverless SQL warehouse capability that admits and routes queries by predicted cost, so a mixed workload of small and large queries queues differently there than on pro or classic.

On a serverless warehouse, incoming queries are scheduled against their predicted resource needs and capacity is brought up ahead of a growing queue, so short dashboard queries are not simply trapped behind a long ad-hoc scan. Pro and classic warehouses have no equivalent management, which is why queuing on those types is addressed by adding clusters, and why moving the workload to serverless can resolve it outright.

Trap That Intelligent Workload Management can be switched on for a pro warehouse, or that it removes the need to set a maximum cluster count on serverless.

1 question tests this

You have an Azure Databricks workspace that contains a serverless SQL warehouse named Warehouse1 sized Medium. Its Scaling setting was left at the default of a minimum and a maximum of one cluster. Wa

The minimum cluster count is what holds warm capacity ready for a predictable peak, while the maximum only permits growth after queries have already started queuing.

Scale-out reacts to observed queuing, so a warehouse that idles overnight and is hit by the whole analyst team at the start of the business day still makes the first users wait however high the maximum is set. Raising the minimum keeps clusters running through the quiet period at a real cost, which is the trade to state explicitly; the maximum remains only the ceiling on how far a busy warehouse may grow.

Trap That raising the maximum cluster count removes the first-thing-in-the-morning latency spike, or that the minimum is purely a cost control with no effect on responsiveness.

2 questions test this

A cluster-scoped init script must live in a Unity Catalog volume, a workspace file, or cloud object storage, and on standard access mode its source must also be allowlisted before the cluster will start.

Init scripts held in DBFS and the legacy global and cluster-named locations are no longer supported, so a Unity Catalog volume is the durable place to keep them. On standard access mode a metastore administrator must add that source to the allowlist as well; without it the cluster fails to start even though the script itself is correct and the user can read the file.

Trap That an init script kept in DBFS still runs, or that read access to the script's location is by itself enough for a standard-access-mode cluster to execute it.

2 questions test this

Node-level operating-system setup belongs in a cluster-scoped init script, engine behavior belongs in a Spark configuration property, and a value a process must read from its own environment belongs in an environment variable.

An init script runs on every node as it joins the cluster, including nodes added later by autoscaling, which is what makes it the mechanism for installing an operating-system package or agent; a shell command typed into a notebook reaches only the driver at that moment and is absent from every node added afterwards. Spark configuration properties change how the engine executes the workload, while environment variables expose values to the driver and executor processes and to the init script itself.

Trap That a shell command run once from a notebook applies to the whole cluster, including the nodes autoscaling adds later.

2 questions test this

A credential supplied through a Spark configuration property or an environment variable must be given as a reference to a secret scope, never as a literal value.

A secret reference is resolved when the compute starts and the value is redacted wherever the configuration is displayed or logged, so the credential never sits in the cluster definition, a policy, or an exported configuration. A key pasted in literally is readable by anyone who can view that compute configuration, which is why the secret scope is the required answer even though the literal value would work.

Trap That a Spark configuration property is private to the cluster, so a literal key is acceptable as long as only the team can attach to that compute.

2 questions test this

The cost-saving capacity model for classic compute is spot workers with an on-demand driver, because a reclaimed worker's tasks are retried elsewhere while losing the driver ends the entire run.

The driver holds the application state and coordinates every executor, so its eviction terminates the cluster and whatever was running on it, whereas a reclaimed spot worker costs only the work in flight on that node. Configuring the driver as on-demand and letting the workers take spot capacity is what makes a cluster cheaper without making it unable to finish.

Trap That placing every node including the driver on spot capacity simply maximizes the discount, since Spark recovers from any node it loses.

3 questions test this

Spot capacity is chosen for interruption-tolerant work such as development, exploration, and restartable batch, and rejected for a deadline-bound run whose lost progress would cost more than the discount.

An eviction forces Spark to recompute the work that node was doing, and on a long shuffle-heavy stage that recomputation can outweigh the hourly saving and push the job past its window. A production run under a delivery commitment therefore belongs on on-demand capacity, or on spot configured to fall back to on-demand so the cluster still acquires nodes when spot capacity is withdrawn.

Trap That spot is always the cheaper choice because its hourly rate is lower, whatever the length or deadline of the job.

2 questions test this

Clusters that repeatedly fail to launch or scale in a capacity-constrained region are answered by enabling flexible node types, which lets Databricks substitute a comparable instance size instead of waiting for the one requested.

Pinning a single instance type ties the cluster's availability to that type's spare capacity in the region, so launches and autoscaling attempts fail while similar families sit free. Flexible node types allow an equivalent alternative to be used automatically, and that availability lever is what a launch-failure scenario is testing.

Trap That a cluster failing to launch in a busy region needs a larger node type, a higher autoscaling maximum, or an instance pool built from that same pinned instance type.

1 question tests this

A workspace admin enabled the Enable auto flexible node types setting in an Azure Databricks workspace last week. Scheduled jobs that run on classic job compute now launch reliably, but a long-lived a

Also tested in

AI-300 Operationalizing Machine Learning and Generative AI Solutions

Match the compute to the workload

Size a cluster: workers, nodes, and autoscaling

Spend less without losing the work

SQL warehouses: size, scale, and Auto Stop

Choose a Databricks Runtime version

Manage cluster-scoped and notebook-scoped libraries

Govern who can create and attach compute

Exam-pattern recognition

Choosing among job, all-purpose, SQL warehouse, and serverless compute

Decision tree

Cheat sheet

Also tested in

References