Study Guide · DP-750

DP-750 Cheat Sheet

380 entries · 12 chapters · 4 domains

Set up and configure an Azure Databricks environment

Select and Configure Compute in a Workspace

Read full chapter

Cheat sheet

Sharp facts the exam loves — scan these before test day.

Job compute is dedicated, ephemeral compute for scheduled production runs

Job (classic jobs) compute provisions a dedicated cluster scoped to a single job run and terminates it when the run finishes, isolating scheduled non-interactive pipelines from interactive development and billing at the lower jobs DBU rate.

Trap All-purpose compute is interactive and shared, so it is not recommended for production jobs and breaks the isolation requirement.

10 questions test this
All-purpose (shared) compute is for interactive, collaborative development

All-purpose compute is persistent, multi-user interactive compute for notebooks and ad hoc analysis; it bills at a higher DBU rate than job compute and is not recommended for production pipelines because it mixes development and production workloads.

10 questions test this
SQL warehouses run SQL and BI workloads, not pipeline code

A SQL warehouse is compute optimized for SQL queries and BI tools such as Power BI and Tableau, and comes in serverless, pro, and classic types; it cannot execute notebook or declarative-pipeline code.

Trap A SQL warehouse cannot serve as the compute engine for a Lakeflow declarative pipeline.

11 questions test this
Serverless compute starts instantly and is managed by Databricks

Serverless compute runs in the Databricks-managed cloud account and starts in seconds with no VM configuration or instance-pool management; it is available for notebooks, jobs, and declarative pipelines.

7 questions test this
Autoscaling adds and removes only worker nodes between a min and max

Cluster autoscaling automatically adds or removes worker nodes between a configured minimum and maximum based on load; it resizes only workers, never the driver, and it does not stop an idle cluster.

Trap Autoscaling only resizes an active cluster; it never terminates an idle one, which is the job of automatic termination.

13 questions test this
A single-node cluster has zero workers and runs on the driver only

A single-node cluster sets the worker count to 0 so the driver runs all Spark work with no worker nodes; it cannot autoscale and suits only small, non-distributed workloads.

4 questions test this
Enhanced autoscaling optimizes Lakeflow declarative-pipeline compute

Lakeflow Spark Declarative Pipelines run on runtime-managed job compute that uses enhanced autoscaling, which scales workers to workload volume and proactively shuts down under-utilized nodes to minimize cost.

3 questions test this
Cluster capacity is worker count multiplied by per-node CPU and memory

Total cluster compute equals the number of worker nodes multiplied by each node's CPU and memory, so a workload is scaled either by adding workers (node count) or by selecting a larger node type.

SQL warehouse size sets query power; scaling adds clusters; Auto Stop halts idle

A SQL warehouse t-shirt cluster size (X-Small to 4X-Large) sets the compute power for a single query, while its Scaling min/max range adds or removes WHOLE clusters to absorb concurrent-query load (not worker nodes, unlike cluster autoscaling); Auto Stop shuts an idle warehouse down (default 45 minutes classic/pro, 10 minutes serverless).

Trap SQL warehouse Scaling adds whole CLUSTERS for concurrency, whereas cluster autoscaling adds WORKER NODES; Auto Stop (warehouse) is distinct from a cluster automatic termination.

5 questions test this
Automatic termination shuts down a cluster after idle minutes

Automatic termination stops an all-purpose cluster after a set number of minutes with no Spark jobs, queries, or commands running, removing idle DBU and VM cost; a cluster is inactive only when nothing is executing, so active workloads keep it alive and are never interrupted.

Trap Enabling autoscaling or spot pricing lowers cost but does not stop an idle cluster; only automatic termination removes the idle waste.

13 questions test this
Instance pools keep warm VMs to reduce cluster start and scale time

An instance pool holds a set of idle, ready-to-use cloud VMs so clusters and autoscaling attach pre-acquired nodes and start faster; you pay the Azure VM cost for idle pooled instances but no DBU until a cluster uses them.

13 questions test this
Spot VMs reduce worker cost but can be reclaimed

Configuring workers as Azure Spot VMs lowers per-hour compute cost, but Azure can reclaim spot capacity at any time, so spot fits fault-tolerant workloads and does not by itself reduce idle-cluster waste.

Photon is the vectorized engine that accelerates SQL and DataFrame work

Photon is Databricks' native vectorized C++ query engine that transparently accelerates SQL and Spark DataFrame workloads, including JSON ETL and Delta operations; it is turned on per cluster with the Use Photon Acceleration checkbox and is on by default on recent runtimes.

Trap Photon accelerates SQL and DataFrame operations, not arbitrary Python or Scala UDFs.

6 questions test this
The Databricks Runtime version fixes the bundled Spark version

Choosing a Databricks Runtime (DBR) version sets the bundled Apache Spark version and preinstalled libraries; Long Term Support (LTS) releases are recommended for production because they are supported and patched longer.

6 questions test this
Databricks Runtime for Machine Learning preinstalls ML libraries

The Databricks Runtime for Machine Learning extends the standard runtime with preinstalled ML libraries such as PyTorch, TensorFlow, and scikit-learn (with GPU variants), so selecting it avoids manually installing the ML stack.

5 questions test this
Cluster-scoped libraries load for every notebook; notebook-scoped are session only

Cluster-scoped libraries install on the cluster and are available to every notebook attached to it, whereas notebook-scoped libraries installed with %pip apply only to the current notebook session and are discarded when the cluster restarts.

9 questions test this
Libraries install from package repositories or from files and volumes

Compute libraries can be installed from package repositories (PyPI, Maven, CRAN) or from uploaded artifacts such as wheel, egg, or JAR files stored in workspace files or Unity Catalog volumes.

8 questions test this
Standard access mode installs only allowlisted libraries

On standard (formerly shared) access mode compute, libraries and init scripts must be on the Unity Catalog allowlist maintained by a metastore admin, whereas dedicated access mode has no such restriction.

Trap The Unity Catalog allowlist governs standard access mode; dedicated (single-user) compute is not subject to it.

5 questions test this
Compute ACLs grant CAN ATTACH TO, CAN RESTART, or CAN MANAGE

Compute access control assigns three permission levels to users, groups, or service principals: CAN ATTACH TO (run commands and view logs), CAN RESTART (start, stop, and restart), and CAN MANAGE (edit configuration and permissions).

14 questions test this
Access mode (standard vs dedicated) sets sharing and Unity Catalog support

A cluster's access mode determines sharing: standard access mode supports multiple concurrent users with full Unity Catalog governance, while dedicated access mode is assigned to a single user or group; both are Unity Catalog-enabled.

Trap The legacy No Isolation Shared access mode is not Unity Catalog-enabled.

6 questions test this
Creating compute requires the cluster-creation entitlement

A user can create all-purpose compute only if granted the unrestricted or policy-scoped cluster-creation entitlement; without it they can merely attach to existing compute they have been given access to.

Cluster policies constrain what compute a user can create

A cluster (compute) policy is an admin-defined rule set that limits the compute a user may create or configure - allowed node types and Databricks Runtime versions, enforced auto-termination, maximum workers or DBU caps, and mandatory tags. Granting a user a policy scopes their cluster-creation entitlement to compute that complies with it, enforcing cost control and standards.

Trap A cluster policy governs the compute a user may CREATE (its configuration limits); that is different from a cluster ACL that grants Can Attach/Can Manage on an already-existing cluster.

15 questions test this
The Azure Databricks disk cache stores copies of remote Parquet files as local files on the worker's SSD, so a repeatedly re-read data set is served by choosing a worker type with local SSD volumes rather than by adding memory

Disk caching copies remote Parquet data files (including Delta Lake tables) into the worker node's local storage on the first read, automatically and with no code change, and later reads of the same files are served locally. The cache is stored as local files on the worker node and is enabled and configured by default on worker types that ship with SSD volumes, using a bounded share of that local SSD space. This is a different mechanism from the Apache Spark cache, which holds in-memory blocks, must be triggered manually in code, and is not what an analytical or ML workload that repeatedly rescans the same tables relies on; in SQL warehouses and on current Databricks Runtimes the CACHE SELECT command is ignored in favor of the automatic disk cache.

Trap Believing the disk cache is an in-memory cache, and therefore selecting a memory-optimized worker type (or running CACHE SELECT / .persist() to prime it) so that a repeatedly re-read data set is cached - when the disk cache is written to the worker's local SSD, is populated automatically on first read, and is only enabled by default on SSD-backed worker types.

The driver node type defaults to the worker node type and must be enlarged separately for driver-side load, because neither more workers nor a higher autoscaling maximum adds driver capacity

In classic compute the driver runs the Spark master, interprets every command, and holds the state of all notebooks attached to the resource, so its instance type is a separate decision from the worker type even though it defaults to the same value. Databricks tells you to pick a larger driver type with more memory when the workload collects a lot of data back from the executors for analysis in the notebook, or when many notebooks stay attached. Autoscaling min and max settings govern worker nodes only, so driver-side pressure is resolved by changing the driver type (or detaching unused notebooks), not by resizing the worker fleet.

Trap Treating a driver out-of-memory failure during a large collect() as a cluster-capacity problem and fixing it by adding workers or raising the autoscaling maximum - the collected result lands on the driver, and worker count and autoscaling bounds never change the driver's instance type or memory.

Classic and pro SQL warehouses admit a fixed number of concurrent queries per cluster, a ratio a larger cluster size does not raise

For classic and pro SQL warehouses the concurrency model fixes how many concurrent queries one cluster admits, so a single cluster takes at most that many at once and everything beyond it waits in a bounded queue. Raising the t-shirt cluster size buys a bigger driver and more workers for each individual query but leaves that per-cluster admission limit untouched, so serving more simultaneous users means raising the maximum number of clusters and letting the warehouse scale out horizontally. A serverless warehouse instead has its workload manager add and release clusters itself as queue wait moves, within the maximum you configure.

Trap Assuming per-cluster query concurrency scales with the cluster size - expecting a bigger t-shirt size to run more simultaneous queries - and answering a growing-user-count problem by sizing up instead of raising the maximum cluster count.

Bytes spilled to disk in the query profile calls for a larger warehouse cluster size, while a persistently non-zero Peak Queued Queries metric calls for more clusters

The two right-sizing signals for a SQL warehouse point at two different settings. Spill reported as Bytes spilled to disk in a query's profile means one query exceeded the memory of the cluster it ran on, and the documented remedy is to increase the cluster size. Peak Queued Queries staying above zero on the warehouse monitoring page means demand exceeds admitted capacity, and the remedy for concurrent load is a higher maximum cluster count; for serverless warehouses Intelligent Workload Management then provisions those extra clusters automatically as queue wait grows.

Trap Reading spill to disk as a capacity shortage and raising the maximum cluster count to fix it - extra clusters only add parallel query slots and give the spilling query no additional memory, because a single query still executes on one cluster.

A shuffle- and memory-heavy transformation that spills is addressed by moving the workers to a memory-optimized instance family, not by adding more workers of the same type.

Wide transformations such as large joins and aggregations hold shuffle data and cached DataFrames in executor memory, and when a node's memory-to-core ratio is too low the stage spills and slows. A memory-optimized worker family raises the memory available per core so those partitions fit, whereas adding nodes of the same type raises total capacity while leaving the per-executor ratio, and therefore the spilling, unchanged.

Trap That a spilling job is fixed by raising the worker count or the autoscaling maximum, on the reasoning that more nodes means more total memory.

2 questions test this
A CPU-bound workload with little shuffle or caching belongs on a compute-optimized worker family, which supplies the cores it needs without paying for memory it will not use.

Work dominated by decoding, parsing, or per-record computation, including many streaming jobs that keep only small state, is limited by cores rather than by memory. A compute-optimized family delivers the most CPU per unit of cost for that profile, while a memory-optimized family chosen for the same core count buys memory the job never touches.

Trap That a memory-optimized family is the safe default for every Spark workload because Spark is an in-memory engine.

1 question tests this
A driver serving many attached notebooks or collecting results back to itself is selected for driver-side memory, so a shared interactive cluster can need a memory-rich driver family while its workers stay on a cheaper one.

All-purpose compute hosts one interpreter and its session state per attached notebook on the driver, and collected results and broadcast tables are assembled there as well, so a cluster shared by many analysts is constrained on the driver long before the workers are. The lever is the driver's instance family, chosen independently of the workers': extra cores on a compute-optimized driver do not hold more state, and a job cluster running a single automated task rarely needs the same treatment.

Trap That the driver should simply mirror whatever family the workers use, so a cluster shared by many notebooks is tuned by changing the worker type instead.

1 question tests this
Serverless notebook and job compute is configured through its environment, an environment version plus declared dependencies, rather than by selecting a Databricks Runtime, attaching a cluster-scoped library, or running an init script.

A serverless workload exposes no cluster specification to edit, so its client image and library baseline come from the serverless environment version that is selected, and extra packages are declared as dependencies of that environment. Where many notebooks and jobs must share one dependency set, a workspace base environment defines it once and is referenced by each workload rather than being installed again per notebook.

Trap That libraries and engine version are pinned on serverless the same way as on classic compute, by choosing a runtime version and attaching cluster-scoped libraries.

2 questions test this
Serverless performance mode is a latency-for-cost lever: performance-optimized buys faster start-up and execution at a higher rate, while standard performance is the cheaper setting for latency-tolerant work.

Both modes run the same code and differ in how aggressively capacity is provisioned for the workload. Interactive notebooks and short user-facing jobs are the case for performance-optimized, while scheduled batch work that nobody is waiting on belongs in standard performance mode so the saving is taken where the delay costs nothing.

Trap That standard performance mode is a smaller machine that risks failing the job, or that performance mode governs how many concurrent workloads are admitted rather than latency and price.

1 question tests this
A stated requirement that serverless does not support, such as a custom init script, a specific node type, spot capacity, or an instance pool, is answered by moving the workload to classic compute rather than by tuning serverless.

Serverless hides the cluster, so every setting that names infrastructure is unavailable by design and cannot be reintroduced through the environment. When a scenario fixes such a requirement the correct answer is classic compute, job compute for scheduled runs and all-purpose for interactive work; when it fixes only outcomes such as immediate start, no idle capacity, and no cluster administration, serverless is the answer.

Trap That any serverless limitation can be worked around inside serverless, for example by supplying an init script or requesting a node type through the environment.

2 questions test this
Warehouse type is the lever for start-up latency and management model: a serverless warehouse becomes available in seconds on Databricks-managed capacity, while pro and classic warehouses run in the customer's own cloud account and take minutes to start.

A dashboard that must not wait on a cold warehouse, or a bursty pattern where capacity should disappear between peaks, is answered by choosing the serverless type, not by enlarging a pro warehouse. Pro is the type to keep when the compute must remain inside the customer's cloud account, and classic is the entry-level type that lacks the newer performance capabilities.

Trap That cold-start delay on a pro or classic warehouse is removed by a larger cluster size or a longer auto-stop timeout.

2 questions test this
Intelligent Workload Management is a serverless SQL warehouse capability that admits and routes queries by predicted cost, so a mixed workload of small and large queries queues differently there than on pro or classic.

On a serverless warehouse, incoming queries are scheduled against their predicted resource needs and capacity is brought up ahead of a growing queue, so short dashboard queries are not simply trapped behind a long ad-hoc scan. Pro and classic warehouses have no equivalent management, which is why queuing on those types is addressed by adding clusters, and why moving the workload to serverless can resolve it outright.

Trap That Intelligent Workload Management can be switched on for a pro warehouse, or that it removes the need to set a maximum cluster count on serverless.

1 question tests this
The minimum cluster count is what holds warm capacity ready for a predictable peak, while the maximum only permits growth after queries have already started queuing.

Scale-out reacts to observed queuing, so a warehouse that idles overnight and is hit by the whole analyst team at the start of the business day still makes the first users wait however high the maximum is set. Raising the minimum keeps clusters running through the quiet period at a real cost, which is the trade to state explicitly; the maximum remains only the ceiling on how far a busy warehouse may grow.

Trap That raising the maximum cluster count removes the first-thing-in-the-morning latency spike, or that the minimum is purely a cost control with no effect on responsiveness.

2 questions test this
A cluster-scoped init script must live in a Unity Catalog volume, a workspace file, or cloud object storage, and on standard access mode its source must also be allowlisted before the cluster will start.

Init scripts held in DBFS and the legacy global and cluster-named locations are no longer supported, so a Unity Catalog volume is the durable place to keep them. On standard access mode a metastore administrator must add that source to the allowlist as well; without it the cluster fails to start even though the script itself is correct and the user can read the file.

Trap That an init script kept in DBFS still runs, or that read access to the script's location is by itself enough for a standard-access-mode cluster to execute it.

2 questions test this
Node-level operating-system setup belongs in a cluster-scoped init script, engine behavior belongs in a Spark configuration property, and a value a process must read from its own environment belongs in an environment variable.

An init script runs on every node as it joins the cluster, including nodes added later by autoscaling, which is what makes it the mechanism for installing an operating-system package or agent; a shell command typed into a notebook reaches only the driver at that moment and is absent from every node added afterwards. Spark configuration properties change how the engine executes the workload, while environment variables expose values to the driver and executor processes and to the init script itself.

Trap That a shell command run once from a notebook applies to the whole cluster, including the nodes autoscaling adds later.

2 questions test this
A credential supplied through a Spark configuration property or an environment variable must be given as a reference to a secret scope, never as a literal value.

A secret reference is resolved when the compute starts and the value is redacted wherever the configuration is displayed or logged, so the credential never sits in the cluster definition, a policy, or an exported configuration. A key pasted in literally is readable by anyone who can view that compute configuration, which is why the secret scope is the required answer even though the literal value would work.

Trap That a Spark configuration property is private to the cluster, so a literal key is acceptable as long as only the team can attach to that compute.

2 questions test this
The cost-saving capacity model for classic compute is spot workers with an on-demand driver, because a reclaimed worker's tasks are retried elsewhere while losing the driver ends the entire run.

The driver holds the application state and coordinates every executor, so its eviction terminates the cluster and whatever was running on it, whereas a reclaimed spot worker costs only the work in flight on that node. Configuring the driver as on-demand and letting the workers take spot capacity is what makes a cluster cheaper without making it unable to finish.

Trap That placing every node including the driver on spot capacity simply maximizes the discount, since Spark recovers from any node it loses.

3 questions test this
Spot capacity is chosen for interruption-tolerant work such as development, exploration, and restartable batch, and rejected for a deadline-bound run whose lost progress would cost more than the discount.

An eviction forces Spark to recompute the work that node was doing, and on a long shuffle-heavy stage that recomputation can outweigh the hourly saving and push the job past its window. A production run under a delivery commitment therefore belongs on on-demand capacity, or on spot configured to fall back to on-demand so the cluster still acquires nodes when spot capacity is withdrawn.

Trap That spot is always the cheaper choice because its hourly rate is lower, whatever the length or deadline of the job.

2 questions test this
Clusters that repeatedly fail to launch or scale in a capacity-constrained region are answered by enabling flexible node types, which lets Databricks substitute a comparable instance size instead of waiting for the one requested.

Pinning a single instance type ties the cluster's availability to that type's spare capacity in the region, so launches and autoscaling attempts fail while similar families sit free. Flexible node types allow an equivalent alternative to be used automatically, and that availability lever is what a launch-failure scenario is testing.

Trap That a cluster failing to launch in a busy region needs a larger node type, a higher autoscaling maximum, or an instance pool built from that same pinned instance type.

1 question tests this

Create and Organize Objects in Unity Catalog

Read full chapter

Cheat sheet

Sharp facts the exam loves — scan these before test day.

Unity Catalog addresses data as catalog.schema.object

Unity Catalog organizes every data object in a three-level namespace of catalog.schema.object (tables, views, volumes, functions, and models), replacing the legacy two-level hive_metastore.schema.table layout.

5 questions test this
Use a separate catalog per environment for data isolation

The catalog is the primary unit of data isolation in Unity Catalog, so a common naming convention creates a distinct catalog per environment (for example dev, test, and prod) to segregate data and permissions.

Trap Separating environments only by schema inside one shared catalog weakens the isolation boundary.

6 questions test this
Bind a catalog to specific workspaces to restrict its access

Workspace-catalog binding restricts a catalog so it is accessible only from designated workspaces, enforcing environment isolation and controlled external sharing across a metastore shared by several workspaces.

5 questions test this
CREATE CATALOG makes a top-level container in the metastore

CREATE CATALOG creates the top-level container within the metastore; an optional MANAGED LOCATION sets where its managed tables and volumes store data, otherwise they inherit managed storage from the metastore.

Trap CREATE CATALOG builds a catalog, not a schema; a contained schema needs CREATE SCHEMA catalog.schema.

9 questions test this
Creating a catalog requires the CREATE CATALOG metastore privilege

Only a metastore admin or a principal granted the CREATE CATALOG privilege on the metastore can create a catalog, and the creator becomes its owner with full control over the new object.

Set a schema's storage with MANAGED LOCATION, not LOCATION, in Unity Catalog

CREATE SCHEMA catalog.schema MANAGED LOCATION '' sets a schema's managed storage in Unity Catalog; the LOCATION clause is a Hive-metastore-only syntax and is rejected for Unity Catalog schemas.

Trap LOCATION is not supported for a Unity Catalog schema; use MANAGED LOCATION instead.

6 questions test this
A schema is created two-level as catalog.schema

A schema (database) groups tables, views, and volumes and must be created inside a catalog with the two-level name catalog.schema; it cannot be created as a top-level object.

7 questions test this
MANAGED LOCATION requires CREATE MANAGED STORAGE on an external location

Setting a schema's MANAGED LOCATION requires the CREATE MANAGED STORAGE privilege on the external location that covers the path; without an explicit managed location the schema inherits managed storage from its catalog or the metastore.

Volumes are Unity Catalog objects that govern non-tabular files

A volume is a Unity Catalog object under a schema that governs access to non-tabular data such as images, CSV files, and model artifacts, accessed through the path /Volumes/catalog/schema/volume.

8 questions test this
Managed volumes use Unity Catalog storage; external volumes point at a location

A managed volume stores its files in the schema's managed storage and is fully lifecycle-managed by Unity Catalog, whereas an external volume registers an existing path under an external location for data that Databricks does not own.

Trap Dropping a managed volume deletes its files; dropping an external volume leaves the files in place.

5 questions test this
A view is a stored read-only query that materializes no data

A view is a saved SELECT query that is evaluated each time it is read; it stores no data of its own and can restrict or reshape the columns exposed from its base tables.

17 questions test this
A materialized view stores precomputed results refreshed incrementally

A materialized view precomputes and stores query results and refreshes them incrementally through a Lakeflow declarative pipeline on serverless compute, speeding repeated reads at the cost of storage and scheduled refresh.

Trap A materialized view stores data and must be refreshed; a standard view does neither.

15 questions test this
A table persists data, defaulting to the Delta Lake format

A Unity Catalog table stores rows and columns using the Delta Lake format by default, and can be created empty, with CREATE TABLE AS SELECT (CTAS), or with CREATE OR REPLACE TABLE.

Lakehouse Federation needs a connection first, then a foreign catalog

To federate an external database you first create a connection object that stores the server host and credentials, then create a foreign catalog that uses that connection to mirror the external database's schemas in Unity Catalog.

9 questions test this
Federated queries run in place with no data copy

A foreign catalog runs Lakehouse Federation queries directly against the source system so its schemas and tables appear alongside other Unity Catalog objects and are queried in place, with no data copied into Databricks-managed storage.

Trap An ingestion pipeline or a managed table would copy the data; only a foreign catalog queries the source in place.

10 questions test this
Dropping a managed table deletes its underlying data

A managed table stores its data files in Unity Catalog managed storage, so DROP TABLE removes both the table metadata and the underlying data files.

Trap Dropping an external table removes only the metadata and leaves the files intact.

8 questions test this
An external table is defined with LOCATION and keeps its data on drop

An external table is created with a LOCATION clause pointing at a path under an external location, so Unity Catalog governs only its metadata and DROP TABLE leaves the underlying files untouched.

11 questions test this
AI/BI Genie answers natural-language questions over a curated data set

An AI/BI Genie Agent (formerly called a Genie space) lets business users ask natural-language questions that Genie converts to SQL over a curated set of Unity Catalog tables, enabling self-service data discovery without writing queries.

10 questions test this
Genie instructions and example queries steer accurate answers

Configuring Genie general instructions, example SQL queries, and verified answers guides how Genie interprets domain terms and business logic, improving the accuracy of the answers it generates for data discovery.

Trap Genie relies on the curated tables plus its instructions, not on unrestricted access to the whole metastore.

10 questions test this
A streaming table refresh applies the current definition only to newly arrived rows, and an incompatible definition change fails the next refresh instead of converting history

A streaming table refresh evaluates only the rows that arrived after the last update and appends them, using the current definition for that new data alone: modifying a streaming table definition does not automatically recalculate existing data. Removing a filter does not reprocess previously filtered rows, changing column projections does not affect how existing data was processed, and a join with a static snapshot uses the snapshot state seen at the time of the initial processing, so late-arriving data that would have matched an updated snapshot is ignored and facts can be dropped when dimensions are late. If a modification is incompatible with existing data - modifying the CAST of an existing column is the documented example - the next refresh fails with an error rather than converting the stored rows.

Trap Expecting a routine refresh to reapply an edited definition to rows that were already processed - assuming that removing a filter backfills the rows it used to exclude, that a changed projection rewrites existing rows, or that widening an existing column's CAST is absorbed quietly instead of failing the refresh.

3 questions test this
A materialized view refreshes incrementally only when its query is incrementalizable and its Delta sources have row tracking enabled; otherwise the system recomputes the whole query

Each materialized view refresh resolves to one of two methods: an incremental refresh that identifies changes since the last update and merges only new or modified rows, or a full refresh that reruns the entire query and replaces the stored results. Incremental refresh is conditional - the source data has to sit in Delta tables with row tracking enabled (ALTER TABLE ... SET TBLPROPERTIES (delta.enableRowTracking = true)), and the query structure has to be incrementalizable, which EXPLAIN CREATE MATERIALIZED VIEW will tell you. By default Databricks applies a cost model and picks whichever method is cheaper for that refresh, and a REFRESH POLICY in the definition overrides that choice; recreating a source table drops row tracking and must be re-enabled.

Trap Assuming that because a materialized view stores precomputed results it always updates incrementally, and so expecting cheap incremental refreshes without enabling row tracking on the Delta sources or checking that the query can be incrementalized.

5 questions test this
The comments on a Unity Catalog function and on its parameters are what tell Genie when to call it and what an argument should look like

Genie decides whether a registered function answers the question in front of it from the function's Unity Catalog metadata, not from its logic, because it cannot see inside the body it is calling. The COMMENT clause on the function is where you describe what the function does and therefore when it applies, and a COMMENT on each parameter is where you describe the value expected for it; Databricks documents precise comments as what lets a tool-calling agent know when and how to use a function, and a comment that only restates the function name is its example of ineffective documentation. Treat the comment as part of the trusted asset rather than as documentation: a function with a vague comment is registered, governed and reachable, and is still passed over in favour of generated SQL.

Trap Answering a function that Genie never calls by adding more general instruction text or another example query, when the metadata that decides selection - the function comment and its parameter comments - says nothing about when the function applies.

3 questions test this
Genie can only call a registered Unity Catalog SQL function with user-supplied parameters - it cannot view or modify the function's SQL - and every user of the Genie Agent needs EXECUTE on that function

A SQL function is the trusted-asset shape for logic too complex for a static or parameterized example query: it lives in Unity Catalog, Genie invokes it with parameter values taken from the user's question, and Genie can neither read nor rewrite the SQL inside it, which is why it suits logic that must not be surfaced or altered. That opacity is also the access-control consequence: users of the Genie Agent must hold the EXECUTE privilege on any SQL function used as a trusted asset, so sharing the Genie Agent alone does not make the function usable. Registering the function once also lets the same certified definition be shared across teams rather than restated as instruction text in every Genie Agent.

Trap Assuming that granting a business user access to the Genie Agent is enough for them to get answers from a function-backed trusted asset, or that Genie will adapt the function's SQL to handle a related question - Genie needs an explicit EXECUTE grant on the function and treats its body as a black box it can only call.

3 questions test this
A managed table or volume is written to the schema's managed storage location if the schema has one, otherwise the catalog's, otherwise the metastore's, so the lowest level that defines a location wins.

Unity Catalog resolves managed storage from the most specific level outward: schema first, then catalog, then metastore. Because a workspace newly enabled for Unity Catalog carries no metastore-level managed storage at all, assuming the data will fall back to the metastore is not safe, and the recommended isolation lever is a managed location on the catalog for each environment rather than one on every schema. Several catalogs and schemas may safely share a single managed location, because Unity Catalog isolates each object's data beneath it.

Trap That managed data always lands in the metastore's root storage, or that a location set on the catalog overrides one already set on the schema.

2 questions test this
A catalog or schema managed location is legal only inside an already-registered external location, while metastore-level managed storage must sit outside external locations and no managed storage may overlap an existing external table or volume path.

The path named as a catalog's or schema's managed location must fall inside storage that is already registered in Unity Catalog as an external location, which is why an otherwise well-formed catalog or schema creation is refused when the container was never registered. Metastore-level managed storage is the exception and must not sit inside an external location, and no managed storage at any level may overlap the path of an existing external table or external volume.

Trap That any storage container the workspace can reach may be named as a managed location, and that overlapping an existing external table's path is untidy rather than rejected.

3 questions test this
Altering a catalog's or schema's managed location governs only managed objects created after the change and physically moves no existing table or volume data.

Managed storage is resolved at the moment an object is created, so changing the location later re-points new managed tables and volumes only; everything already written stays where it first landed. A requirement to relocate data that already exists is therefore answered by recreating or copying those objects into the new location, never by the alter statement on its own.

Trap That setting a new managed location on the catalog or schema migrates the managed tables and volumes that are already there into the new container.

2 questions test this
A shallow clone is the answer when a receiving catalog needs a writable copy that may freely diverge with no upfront data duplication, and a deep clone is the answer when the copy must outlive the source, sit in different storage, or go to a team that will never have access to the source's files.

A shallow clone copies metadata only and references the source table's existing data files, so it completes in seconds against a multi-terabyte table and duplicates no bytes; a deep clone physically copies the data and is therefore the choice when the copy must stand on its own. Writes to either clone leave the source untouched, and re-issuing the same clone against an existing target syncs it incrementally instead of rebuilding it, which is what makes a clone the repeatable environment-refresh answer rather than a one-off query-built copy.

Trap That a development copy of a large production table must be a deep clone or a query-built copy so the team can write to it, and that refreshing that copy means dropping and rebuilding it each time.

2 questions test this
Shallow clone applies to Delta tables only and a clone must be managed-to-managed or external-to-external, so a managed source cannot be shallow cloned into an external target.

Unity Catalog requires the clone target to be the same kind of table as its source, and shallow clone additionally supports Delta tables only. A plan to shallow clone a production managed table into an external table in a development container fails on that rule alone, whatever privileges the requester holds, and has to be restructured as a managed target in the development catalog or as a deep clone.

Trap That any governed table can be shallow cloned to any container the user can write to, so the plan only needs the right privileges on the target.

2 questions test this
Running VACUUM on the source of a Unity Catalog shallow clone does not break the clone, because Unity Catalog tracks which source files the clone still references.

Under the legacy metastore a shallow clone could be orphaned when its base table was vacuumed, and that belief is carried forward into Unity Catalog scenarios where it no longer holds. Unity Catalog tracks the source files a clone still depends on, so the clone keeps reading them, and the same reference tracking is why vacuuming the base table can require access to the clone.

Trap That the shallow clone must be recreated, or promoted to a deep clone, before the source table's retention job runs, because VACUUM will remove the files it points at.

3 questions test this

Secure and govern Unity Catalog objects

Secure Unity Catalog Objects

Read full chapter

Cheat sheet

Sharp facts the exam loves — scan these before test day.

GRANT assigns a privilege on a securable to a user, group, or service principal

Access to a Unity Catalog securable is granted with GRANT ON TO , where the principal can be a user, an account group, or a service principal, and REVOKE removes it; common privileges include SELECT, MODIFY, and CREATE.

Trap A principal is not limited to individual users; groups and service principals are equally valid grant targets and groups are preferred for manageability.

6 questions test this
Reading a table also requires USE CATALOG and USE SCHEMA on its parents

To query a table a principal needs SELECT on the table plus USE CATALOG on its catalog and USE SCHEMA on its schema; the USE privileges grant traversal of the three-level namespace but do not by themselves expose any data.

Trap SELECT on the table alone is insufficient; without USE CATALOG and USE SCHEMA on the parents the query fails with a permission error.

11 questions test this
A privilege granted on a catalog or schema is inherited by its child objects

Unity Catalog privileges are inherited down the object hierarchy, so a privilege granted on a catalog applies to all of its current and future schemas and tables, and a grant on a schema applies to its tables, views, and volumes.

5 questions test this
Object owners and MANAGE control who can grant on a securable

In Unity Catalog the owner of a securable (a user, group, or service principal) implicitly holds all privileges on it and is the principal who can GRANT/REVOKE, ALTER, and DROP it; granting the MANAGE privilege lets a non-owner administer the object, including granting and revoking privileges and even dropping or transferring it, without being the owner, and ownership can be reassigned with ALTER OWNER TO.

Trap Holding SELECT is not enough to administer an object - only the owner or a MANAGE holder can grant, drop, or transfer it, and MANAGE differs from ownership only in that it is not automatically given the object's data privileges (it must self-grant SELECT).

UC principals are account-level identities federated from Microsoft Entra ID

Users, groups, and service principals in Unity Catalog are account-level identities, typically provisioned from Microsoft Entra ID via SCIM; account groups must be assigned to a workspace through identity federation before they can be granted privileges there.

Trap Groups are managed at the account level, not per-workspace - grants target account-level principals federated to the workspace, not a workspace-local group.

Object-level grants cover every column; column limits need a view, mask, or ABAC

A SELECT grant applies to the entire table securable and cannot be scoped to individual columns, so column-level access control is achieved by layering a view that exposes only permitted columns, a column mask, or an ABAC policy on top of the base grant.

Trap Granting SELECT on only the non-sensitive columns is not a Unity Catalog capability; it would break SELECT * with a permission error rather than hide values.

15 questions test this
A dynamic view redacts columns per group with is_account_group_member in CASE

A dynamic view wraps a base table and uses CASE expressions calling is_account_group_member() so that members of an authorized group receive the real column value while all other callers receive NULL or a redacted literal.

13 questions test this
Grant the secure view and withhold the base table so users query only the view

For view-based access control you grant SELECT on the dynamic or restricted view and do not grant access to the underlying base table, forcing every principal through the view where the column and row rules are enforced.

A dynamic view enforces row-level security with a caller-identity WHERE predicate

Row-level security through a dynamic view adds a WHERE predicate built from current_user() or is_account_group_member(), returning only the rows whose values match the querying principal, such as their own region or business unit.

16 questions test this
Data-driven row-level security joins to an entitlement mapping table

A scalable row-level-security pattern joins the base table to a mapping table that records which principal or group may see which key values, so entitlement changes are made by editing data rather than by rewriting the view definition.

An Azure Key Vault-backed secret scope exposes Key Vault secrets read-only

An Azure Key Vault-backed secret scope maps a Databricks secret scope onto an Azure Key Vault so notebooks can read its secrets; it is a read-only interface, meaning the secret values are created, updated, and rotated in Azure rather than in Databricks.

Trap A Key Vault-backed scope cannot be written from Databricks; secrets must be added and rotated in the Azure Key Vault itself.

9 questions test this
dbutils.secrets.get reads a secret and Databricks redacts it from output

Code retrieves a secret with dbutils.secrets.get(scope, key) instead of hardcoding credentials, and Databricks automatically replaces any printed secret value with [REDACTED] so it cannot leak into notebook cell output or logs.

13 questions test this
Secret access is governed by READ, WRITE, and MANAGE scope ACLs

Secret scope access control assigns per-scope ACLs at the READ, WRITE, and MANAGE levels, and a principal needs at least READ on the scope, which permits reading secret values and listing keys, before dbutils.secrets.get will succeed.

A service principal is a non-human identity for automated data workloads

A service principal is an identity created for tools, jobs, and CI/CD rather than a person, and it is granted Unity Catalog privileges like any principal so automated pipelines can authenticate and access data without depending on an individual user's account.

16 questions test this
Running a job as a service principal decouples it from a user account

Configuring a Lakeflow Job or pipeline to run as a service principal keeps it working when the original author leaves or loses access, and confines the job's data access to exactly the Unity Catalog privileges granted to that service principal.

12 questions test this
Service principals authenticate with OAuth machine-to-machine tokens

A service principal authenticates non-interactively using OAuth machine-to-machine (M2M) tokens minted from its client ID and secret, which are the recommended automation credential in place of long-lived personal access tokens.

An Access Connector managed identity authenticates the workspace to Azure storage

Resource access to ADLS Gen2 uses an Access Connector for Azure Databricks, a first-party Azure resource whose system- or user-assigned managed identity is granted a storage role such as Storage Blob Data Contributor, letting Databricks reach the storage account with no stored keys.

Trap A managed identity authenticates the underlying storage resource; a service principal instead represents a principal whose Unity Catalog privileges govern data access, not the storage connection.

14 questions test this
A storage credential wraps the managed identity for an external location to use

In Unity Catalog the Access Connector's managed identity is registered as a storage credential, and an external location then references that credential to govern reads and writes to a specific ADLS Gen2 container path.

9 questions test this
Managed identities are preferred over storage account keys or SAS tokens

Authenticating storage access through an Access Connector managed identity is recommended over embedding storage account keys or SAS tokens, because the credential is managed by Azure and is never exposed in notebook code or cluster configuration.

A securable has exactly one owner, while MANAGE can be granted to many principals

Unity Catalog allows only one owning principal per securable — a user, service principal, or group — and ALTER ... OWNER TO replaces that owner rather than adding one. MANAGE is an ordinary privilege that can be granted to any number of principals, and it confers the ability to grant and revoke privileges on the object, transfer its ownership, rename it and drop it without being the owner. Making a group the owner, or granting MANAGE to a group, is therefore how several people share administration of one object.

Trap Believing you can add co-owners to a table or catalog so that several administrators own it simultaneously, instead of granting MANAGE or owning it through a group.

2 questions test this
ALL PRIVILEGES deliberately excludes MANAGE, READ METADATA, EXTERNAL USE SCHEMA and EXTERNAL USE LOCATION

ALL PRIVILEGES implies every applicable privilege for the object type without granting each one explicitly, but it never includes MANAGE, READ METADATA, EXTERNAL USE SCHEMA or EXTERNAL USE LOCATION — the exclusions exist to prevent accidental privilege escalation and data exfiltration. Consequently a principal with ALL PRIVILEGES on a table can read and write it yet cannot grant anyone else access, and revoking ALL PRIVILEGES removes the implied privileges but leaves those four untouched if they were granted separately. Because ALL PRIVILEGES is evaluated at permission-check time, it automatically picks up newly released privileges for that securable type.

Trap Reading ALL PRIVILEGES as literally every privilege, so that the grantee can also administer grants, transfer ownership, or hand a path to an external engine.

6 questions test this
MANAGE has reduced usage requirements: MANAGE on a catalog needs no USE CATALOG, but data access still does

To exercise MANAGE you need usage privileges only on the container levels strictly ABOVE where MANAGE is held, never at that level itself: MANAGE on a catalog requires no USE CATALOG or USE SCHEMA at all, MANAGE on a schema requires USE CATALOG on the parent catalog, and MANAGE on a table requires USE CATALOG plus USE SCHEMA. MANAGE granted on a container is also inherited by every child object, so MANAGE on a catalog carries MANAGE on its schemas and tables. The reduction applies only to the metadata capabilities of MANAGE — data privileges such as SELECT and MODIFY that a MANAGE holder grants to itself still require USE CATALOG and USE SCHEMA.

Trap Assuming a principal with MANAGE on a catalog must also be granted USE CATALOG before they can administer grants inside it, by analogy with the SELECT + USE CATALOG + USE SCHEMA rule.

1 question tests this
READ METADATA, not BROWSE, is the read-only delegation of MANAGE that exposes grants, filters, masks and policies

READ METADATA is the child privilege of the composite MANAGE privilege: it gives read-only visibility into the same owner-visible metadata — permissions, row filters, column masks, ABAC policies, and credential names and IDs — without any ability to modify the object or read its data, which is what an auditor or SRE needs. BROWSE is a different, discovery-oriented privilege: it lets a principal see that an object exists and view its name, description and tags without USE CATALOG or USE SCHEMA, and it deliberately does not expose that security-sensitive metadata. The two are granted and revoked independently of MANAGE, so revoking MANAGE does not revoke an explicitly granted READ METADATA.

Trap Granting BROWSE to a security auditor who must review who has access and which masks are applied, on the belief that BROWSE is the read-only view of an object's governance metadata.

1 question tests this
Creating an external location requires CREATE EXTERNAL LOCATION on both the metastore and the storage credential it references

CREATE EXTERNAL LOCATION is one of the few privileges that must be held in two places at once: on the Unity Catalog metastore, and on the specific storage credential named in the WITH (STORAGE CREDENTIAL ...) clause. Holding it on the metastore alone is not sufficient, which is what stops any metastore-level creator from wrapping someone else's credential in a new path. Metastore admins and workspace admins have this privilege by default, and creating the storage credential itself is separately gated by CREATE STORAGE CREDENTIAL on the metastore.

Trap Assuming that because an external location is a metastore-level securable, a metastore-level CREATE EXTERNAL LOCATION grant alone lets a principal register a path over any existing storage credential.

CREATE EXTERNAL VOLUME applies only to external locations, so an external volume cannot be created from a bare storage credential

The privileges that apply to a storage credential are ALL PRIVILEGES, CREATE EXTERNAL LOCATION, CREATE EXTERNAL TABLE, MANAGE, READ FILES, READ METADATA and WRITE FILES — CREATE EXTERNAL VOLUME is not among them, because it is defined only for external locations. CREATE EXTERNAL TABLE exists on both securables, and Databricks recommends granting it on the external location rather than the credential precisely because the external location is scoped to a path, giving control over where in the cloud tenant users may create tables. So registering an external volume always starts by creating an external location over the target path.

Trap Believing CREATE EXTERNAL VOLUME can be granted on a storage credential the same way CREATE EXTERNAL TABLE can, since both securables authorize access to cloud storage.

EXTERNAL USE LOCATION is excluded from ALL PRIVILEGES and is not held by external location owners by default

EXTERNAL USE LOCATION is the privilege that lets a principal obtain a temporary credential for an external location so an external processing engine can read the path through the Unity Catalog open APIs. To avoid accidental data exfiltration it is excluded from ALL PRIVILEGES, and external location owners do not have it by default; the documentation states that only users with MANAGE on the external location can grant it. The equivalent carve-out exists one level up in the data hierarchy: schema owners do not hold EXTERNAL USE SCHEMA by default either, and only the catalog owner can grant it.

Trap Assuming that granting ALL PRIVILEGES on an external location, or simply owning it, is enough to let an outside engine obtain temporary credentials for that path.

By default a storage credential or external location is usable from every workspace attached to the metastore until it is workspace-bound

Storage credentials and external locations are metastore-level securables, and by default any privileged user can use them from any workspace attached to that metastore. Restricting them to a subset of workspaces is a separate, explicit step — assigning the object to specific workspaces on its Workspaces tab, also called workspace binding or external location isolation. A second, orthogonal control is the Limit to read-only use option, which can be set on the storage credential (making every external location that uses it read-only) or on an individual external location.

Trap Assuming an external location or storage credential is automatically scoped to the workspace in which it was created, so a principal in another attached workspace cannot use it.

A team that only needs to read or write files under a governed path is granted on the volume, which scopes them to that volume's path instead of the whole storage prefix

READ VOLUME and WRITE VOLUME on a Unity Catalog volume, together with USE CATALOG and USE SCHEMA on its parents, give notebooks, jobs and libraries file access through the volume's /Volumes path, and that is the least-privilege securable for a pure file-access requirement over ADLS Gen2 data. Granting on the external location instead exposes every path beneath it, including data belonging to other teams, and is justified only when the principal must also define new objects on that path.

Trap Reaching for a grant on the external location because the data 'is just files in the storage account', when a grant on the volume already covers reading and writing those files at a much narrower scope.

2 questions test this
Creating external tables, external volumes or managed storage over a cloud path requires a grant on the external location covering that path, which no volume grant can confer

The external location is the securable that governs the storage path itself, so the privileges that let a principal read the raw path or define new objects over it — such as READ FILES, WRITE FILES, CREATE EXTERNAL TABLE and CREATE EXTERNAL VOLUME — are granted on the external location. A volume grant never carries this, because a volume exposes only the files beneath its own path to consumers of that volume. Match the securable to the verb in the requirement: consuming files points at the volume, defining objects on the path points at the external location.

Trap Assuming that because a team can already read the files through a volume they can point an external table at the same path, when the create-time authorization check is made against the external location.

3 questions test this
Granting a principal permissions directly on the ADLS Gen2 account or container bypasses Unity Catalog governance entirely, leaving that access ungoverned and unaudited by the metastore

Unity Catalog privileges, lineage and audit apply only to access that arrives through its securables, so a principal holding storage-level roles on the account, or using a path-based credential, reads the same bytes with none of that oversight and cannot be cut off by revoking a grant. The governed answer is to register the path as an external location or a volume backed by a storage credential and grant on that securable, reserving direct storage permissions for the managed identity Unity Catalog itself authenticates with.

Trap Solving a file-access request by assigning the team a data role on the storage container, on the belief that Unity Catalog will still govern and audit the access because the same path is registered as an external location.

3 questions test this
Compute that cannot enforce a row filter, column mask or dynamic view fails the query rather than returning unprotected rows

Fine-grained access control is fail-closed, so a workload running on compute that does not support it errors out instead of quietly serving unfiltered or unmasked data. That makes the symptom diagnostic: a query that fails only against protected tables while succeeding against unprotected ones points at the compute's access mode, whereas users genuinely seeing values they should not points at the grant, the group membership or the policy definition. Fix the first by moving the workload, never by rewriting a control that is working as designed.

Trap Believing an unsupported cluster silently ignores the filter and leaks the underlying values, so the remedy is to re-apply the mask or re-grant privileges rather than to change the compute.

5 questions test this
Dedicated access mode enforces filters and masks only through serverless data filtering, which requires both a workspace enabled for serverless and support for the specific operation

On dedicated compute the protected read is delegated to a serverless filtering layer, so a supported runtime is only half the prerequisite — the workspace must also be enabled for serverless compute, and upgrading the runtime alone will not make a filtered or masked table readable there. Support is granted per operation as well: reads becoming available on dedicated compute does not mean writes, merges or streaming operations against the same protected table are allowed. A workload that must write to a filtered or masked table therefore belongs on compute that enforces the control natively.

Trap Concluding that a newer runtime is all a dedicated cluster needs, or that because reads of the masked table already succeed there the job's writes to it will succeed too.

5 questions test this
Legacy no-isolation compute cannot reach Unity Catalog data at all, so the remedy is moving the workload to a governed access mode rather than granting more privileges

No-isolation clusters sit outside Unity Catalog's identity model, so no grant, runtime upgrade or policy change lets them query a Unity Catalog table, protected or not. When a job on such a cluster fails against the metastore, the fix is to re-run it on standard access mode, serverless compute or a SQL warehouse. Standard access mode is the default landing place when several users share the compute and fine-grained controls must be enforced natively for each caller's identity.

Trap Reading the failure as a missing privilege and adding SELECT plus USE CATALOG and USE SCHEMA, or bumping the runtime version, expecting the no-isolation cluster to then reach the table.

3 questions test this

Govern Unity Catalog Objects

Read full chapter

Cheat sheet

Sharp facts the exam loves — scan these before test day.

Table and column descriptions are added with the COMMENT clause

Descriptions for discovery are set with the COMMENT clause or COMMENT ON and can be edited in Catalog Explorer; a table comment documents the dataset and per-column comments document each field, all stored as Unity Catalog metadata.

10 questions test this
AI-generated comments propose table and column descriptions for review

Catalog Explorer can suggest AI-generated table and column descriptions that a data steward reviews and accepts, accelerating the documentation of large catalogs so that objects become discoverable more quickly.

5 questions test this
Descriptions persist as metadata and power search and discovery

Because comments persist as Unity Catalog metadata they survive schema evolution and surface in Catalog Explorer search and AI/BI Genie, letting users find and understand data without opening the underlying files.

Governed tags are the account-level attribute vocabulary ABAC builds on

Governed tags are a centrally defined, account-level set of tag keys and allowed values, with permissions controlling who may apply each tag; they are the attributes that attribute-based access control policies evaluate to decide protection.

10 questions test this
An ABAC policy applies filters or masks automatically to every tag-matched object

An attribute-based access control (ABAC) policy is attached at a catalog, schema, or table and uses governed-tag conditions to apply a row filter or column mask to every current and future object carrying the matching tag, so one policy governs many tables at once.

Trap ABAC scales a single tag-driven policy across many tables; a table-level SET MASK or SET ROW FILTER must be configured on each table individually.

10 questions test this
Tags assigned to a parent object are inherited by child objects

A governed tag applied to a catalog or schema is inherited by the schemas and tables beneath it, but not by individual table columns, so tagging a parent propagates the attribute that ABAC policies and discovery rely on to descendant tables; a column-level classification must be tagged on the column directly.

A row filter removes whole rows via ALTER TABLE ... SET ROW FILTER

A row filter is a boolean SQL UDF attached with ALTER TABLE SET ROW FILTER; the function is evaluated per row and returns TRUE to keep the row or FALSE/NULL to drop it, so it controls which entire rows a principal sees at query time.

Trap A row filter removes whole rows and cannot reveal only part of a column value; use a column mask for partial redaction.

19 questions test this
A column mask redacts values via ALTER TABLE ... ALTER COLUMN ... SET MASK

A column mask is a SQL UDF attached with ALTER TABLE ALTER COLUMN SET MASK; it rewrites each returned value at query time, for example showing only the text after the @ in an email or only the last four digits of a card, while every row is still returned.

Trap Masking only sensitive columns while keeping table SELECT follows least privilege and lets queries run without errors, unlike revoking column access.

19 questions test this
Row filters and column masks attach to base tables, not to standard views

Table-level row filters and column masks are attached to base tables and cannot be placed on a standard view, while ABAC policies extend the same row-filter and column-mask protection to materialized views and streaming tables; once attached they are enforced for every query through any SQL warehouse, notebook, or client.

delta.deletedFileRetentionDuration sets the VACUUM retention window

The table property delta.deletedFileRetentionDuration defines how long removed data files are retained before VACUUM is permitted to delete them, defaulting to 7 days; raising it lengthens the window during which older versions stay recoverable.

18 questions test this
delta.logRetentionDuration bounds how far back time travel can go

The table property delta.logRetentionDuration controls how long transaction-log history is kept, defaulting to 30 days, which bounds the versions and timestamps that time-travel queries can reference.

18 questions test this
Running VACUUM permanently deletes old files and forfeits earlier time travel

VACUUM permanently removes data files that are no longer referenced by the latest table state and are older than the retention threshold, after which you can no longer time-travel to a version whose files were purged, trading storage cost against recoverability.

Unity Catalog captures lineage automatically down to the column level

For queries and workflows run on Azure Databricks, Unity Catalog captures runtime data lineage automatically down to the column level with no configuration, and aggregates it across every workspace attached to the metastore.

10 questions test this
Catalog Explorer shows owner, history, dependencies, and upstream/downstream lineage

The Catalog Explorer Lineage tab renders an interactive graph of upstream and downstream tables, columns, notebooks, jobs, and dashboards alongside the object owner and history, so you can trace dependencies before changing or deleting an object.

13 questions test this
Unity Catalog audit events are queryable in the system.access.audit table

The system.access.audit system table records account- and workspace-level audit events, capturing which principal accessed which securable and what action was taken, and retains them for up to one year for security and compliance analysis.

7 questions test this
Azure diagnostic settings deliver Databricks audit logs to Log Analytics

Configuring Azure diagnostic settings on the workspace streams Azure Databricks diagnostic (audit) logs to a Log Analytics workspace, a storage account, or Event Hubs for long-term retention, querying, and alerting in Azure Monitor.

6 questions test this
Verbose audit logging adds notebook and SQL command events

Enabling verbose audit logging records additional fine-grained events such as commandSubmit, commandFinish, and runCommand, capturing notebook-cell and SQL-warehouse command activity that standard audit logging omits.

Databricks-to-Databricks sharing needs no token when the recipient has Unity Catalog

When the recipient also has a Unity Catalog-enabled workspace, a recipient of authentication type DATABRICKS shares data over a secure Databricks-managed channel identified by a sharing identifier, so no bearer token is created or managed and identity, authentication, and auditing are handled by the platform.

Trap Databricks-to-Databricks sharing eliminates token management; only open sharing to a non-Databricks recipient issues a bearer token.

12 questions test this
Open sharing uses a bearer token and credential file for non-Databricks recipients

Open sharing (the Databricks-to-Open protocol) reaches recipients on any platform by authenticating a recipient of type TOKEN using a long-lived bearer token or OIDC (OpenID Connect) federation, where the recipient presents a short-lived token minted by its own identity provider instead of a stored Databricks credential; for a token, Databricks generates a credential file delivered via an activation link that must be secured and rotated.

14 questions test this
A secure share exposes only selected tables to a named recipient

A secure OpenSharing (previously Delta Sharing) strategy creates a share, adds only the specific tables or views to be exposed, and grants that share to a defined recipient, so the recipient receives read-only access to just the shared objects rather than to the whole metastore. OpenSharing names the feature; its two recipient modes are Databricks-to-Databricks sharing, which needs no token, and open sharing (the Databricks-to-Open protocol), which issues one.

A share partition filter written against CURRENT_RECIPIENT lets one share deliver a different row slice to each recipient

When a table is added to a share you can supply a partition specification that references a recipient property instead of a literal, for example ALTER SHARE acme ADD TABLE acme.default.some_table PARTITION (country = CURRENT_RECIPIENT().country). Databricks then delivers to each recipient only the rows whose column value equals that recipient's property value, so the same share and the same table can be granted to many recipients across different accounts, workspaces and metastores while data boundaries are preserved. The documentation states that without this parameterized partition sharing you would have to create a separate share for each recipient.

Trap Believing that giving two partners different row subsets of the same table requires two shares (or two derived tables), because a share is a static list of objects and cannot be evaluated per recipient.

3 questions test this
The predefined databricks.accountId and databricks.metastoreId recipient properties exist only for Databricks-to-Databricks recipients

Every recipient object carries predefined properties that begin with 'databricks.': databricks.accountId and databricks.metastoreId identify the recipient's Databricks account and Unity Catalog metastore and are documented as Databricks-to-Databricks sharing ONLY, while databricks.name is simply the recipient's name and is the property surfaced for open (token-authenticated) recipients. Any additional key you need for filtering must be created as a custom property, either at creation time with CREATE RECIPIENT ... PROPERTIES ('country' = 'us') or afterwards with ALTER RECIPIENT ... SET PROPERTIES / UNSET PROPERTIES. So a design that partitions shared data by Databricks account ID works only when every recipient is on Unity Catalog; an open recipient needs a custom property instead.

Trap Assuming databricks.accountId or databricks.metastoreId can be used to partition data for a non-Databricks (bearer-token / open sharing) recipient, since the recipient object exists in both sharing modes.

1 question tests this
Per-recipient COLUMN redaction requires a shared dynamic view; a share partition filter can only include or exclude whole rows

A share partition filter expresses a single equality between a table column and a recipient property, so it can withhold rows but cannot alter the value a recipient sees in a column. To vary column values per recipient the provider creates a view whose definition calls CURRENT_RECIPIENT('') inside a CASE expression — for example returning the pii column when CURRENT_RECIPIENT('country') = 'US' and the literal 'REDACTED' otherwise — and then adds that view to the share exactly as a normal view. The same function also supports row-level predicates in a view (WHERE country = CURRENT_RECIPIENT('country')), and providers cannot create another view that references a dynamic view.

Trap Believing a share partition filter can mask or redact a sensitive column per recipient, rather than only filtering which rows are delivered.

3 questions test this
A provider cannot validate a CURRENT_RECIPIENT view by querying it directly, because outside a sharing context the function has no value to return

A view that calls CURRENT_RECIPIENT is for sharing only: when the provider selects from it in their own workspace the function evaluation fails for lack of a sharing context, so the query errors rather than returning unfiltered data. The two documented ways to test it are to mock the context in the session with SET RECIPIENT , which sets CURRENT_RECIPIENT for that session, or to share the view with yourself and query it as a recipient.

Trap Assuming the provider can smoke-test a dynamic view by simply running SELECT * on it before sharing, and that the result would show the unfiltered base rows.

1 question tests this
Row filter and column mask UDFs run with definer's rights, except user-context functions which evaluate as the invoker

The documentation states that all filters run with definer's rights except for functions that check user context — for example SESSION_USER and IS_ACCOUNT_GROUP_MEMBER — which run as the invoker. That combination is what makes the mapping-table (access-control-list) pattern work: a filter such as RETURN EXISTS(SELECT 1 FROM valid_users v WHERE v.username = SESSION_USER()) reads the entitlement table under the function definer's rights while still resolving the identity of whoever ran the query, so the querying user does not need to be granted access to the entitlement table itself.

Trap Concluding that a mapping-table-driven row filter fails unless every end user is also granted SELECT on the mapping table, since the filter reads that table during their query.

2 questions test this
A UDF parameter type that does not match its column is implicitly cast, and with ANSI mode off uncastable values become NULL, silently defeating the filter

The data type of each table column passed to a row filter or column mask must match the corresponding UDF parameter type; on a mismatch Databricks implicitly casts the column value. With spark.sql.ansi.enabled = false, values that cannot be cast are converted to NULL with no error raised, so a filter written as RETURN dept IS NULL over a STRING column bound to an INT parameter evaluates to true for every row and the query returns the entire table. Databricks recommends enabling ANSI mode so a failed cast raises an error and the defect is visible instead of silently returning the wrong rows or masking the wrong values.

Trap Assuming a type mismatch between the column and the UDF parameter is always caught at ALTER TABLE time or raises an error at query time, so a filter can never silently fail open.

3 questions test this
Reading a filtered or masked table requires a SQL warehouse, standard access mode, or dedicated access mode on a recent LTS runtime or serverless compute with serverless enabled

To query a table carrying a row filter or a column mask the compute must be a SQL warehouse, standard access mode on a supported Databricks Runtime, or dedicated access mode on a more recent long-term-support runtime - and dedicated access mode additionally requires the workspace to be enabled for serverless compute, because the fine-grained access control that enforces the policy runs on serverless and can incur serverless charges. Dedicated compute on a runtime below that floor cannot read such a table at all, writing from dedicated compute needs a newer runtime still, and a runtime below the supported floor fails SECURELY: it returns no data rather than returning the table unfiltered.

Trap Believing that because Unity Catalog enforces the policy centrally, any Unity Catalog-enabled cluster can read the table - and that an unsupported runtime would fall back to showing the unfiltered rows.

2 questions test this
Dropping the filter or mask UDF before detaching it from the table leaves the table in an inaccessible state

The policy must be removed from the table first — ALTER TABLE DROP ROW FILTER, or ALTER TABLE ALTER COLUMN DROP MASK — and only then DROP FUNCTION. If the function is dropped first the table is left holding an orphaned policy reference and becomes inaccessible; recovery is to run the same ALTER TABLE ... DROP ROW FILTER / DROP MASK statement to clear the dangling reference. To change the logic without any of this, use CREATE OR REPLACE FUNCTION, which leaves the attachment intact.

Trap Expecting that dropping the masking or filtering UDF automatically detaches the policy and returns the table to unfiltered, fully readable behaviour.

1 question tests this
Data Classification is enabled per catalog by its owner or a MANAGE holder, and only a future-inclusive schema scope keeps schemas created later under scan

A catalog owner, or a principal holding MANAGE on the catalog, turns Data Classification on for that catalog, which starts an incremental background scan of its tables and records each detection as a system governed tag for the sensitive class found. The schema scope chosen at enablement decides what happens next: a scope covering selected and future schemas keeps every schema added to the catalog afterwards in the scan, while a scope limited to only the schemas selected at that moment leaves every later schema unscanned until someone reopens the configuration. Choose the future-inclusive scope whenever the requirement is that newly created data be discovered without ongoing manual configuration, and remember that enabling classification on one catalog says nothing about other catalogs in the metastore.

Trap Believing that once Data Classification is enabled on a catalog everything beneath it is permanently covered, so schemas created after enablement are scanned regardless of which schema scope was selected.

2 questions test this
Data Classification finds and labels sensitive columns but does not restrict access to them, so a requirement that a group must not see the values still needs a masking control

The output of a classification scan is a governed tag recording that a sensitive class was detected in a column; the tag changes discovery and policy targeting, not who can read the data. A requirement that a group must not see sensitive values is still met by a masking control — an attribute-based access control policy, a column mask, or a secure view — with classification supplying the tags that control matches on. Read the requirement in the other direction too: classification is the right answer when newly created tables must be found without anyone hand-building an inventory of sensitive columns, and views and metric views are never scanned, so a view over sensitive data is addressed by classifying its underlying tables.

Trap Treating enablement of the scan as the access control itself — answering that turning Data Classification on stops an analyst group from reading the columns it detected as PII.

2 questions test this
Automatic tagging of detected columns is a separate switch from classification, with its own privilege set and a catalog setting that overrides the metastore default

Starting the scan and applying the detected class tags are two different settings: tagging additionally requires USE CATALOG and APPLY TAG on the catalog plus ASSIGN on the tag being applied, and the classification system tags are account-admin-controlled by default, so a data steward who can enable the scan may still be unable to enable tagging. A catalog-level tagging setting overrides the metastore-level default, which is how one catalog can tag automatically while the rest of the metastore does not. Turning tagging on tags existing detections on the next scan rather than backfilling them at once, and turning it off stops future tags without removing the tags already applied.

Trap Assuming whoever can enable Data Classification on a catalog can also enable automatic tagging, and that flipping tagging on immediately tags every column already detected while flipping it off removes them.

3 questions test this
Notebook files, Unity Catalog volumes and Unity Catalog models travel only over the Databricks-to-Databricks protocol, so a recipient without Unity Catalog can be given tabular assets only

What a share may carry is decided by the recipient's protocol, not by how the share is built: a recipient on their own Unity Catalog metastore can receive notebooks, volumes and models alongside tables and views, while an open-protocol recipient reached through a credential file can receive tabular assets only. A requirement to hand a non-Databricks partner raw files or a notebook is therefore not satisfied by adding those assets to the share and needs a different delivery path. Views, materialized views and streaming tables are shareable, but an open recipient receives only the current snapshot with no time travel, streaming read or change data feed, and the provider materializes and filters those assets at its own cost — which is what makes a curated view an expensive answer to a cheap requirement.

Trap Believing a volume or a notebook added to the share will reach any recipient once they hold the credential file, since a credential file is all an external partner needs to read shared tables.

3 questions test this
Sharing a table WITH HISTORY is what lets the recipient time travel, stream from it and run transactions, and it is mandatory for tables using deletion vectors or column mapping

The history option is chosen from what the consumer must be able to do: without history the recipient sees the current state only, and with history they can query earlier versions, consume the table as a Structured Streaming source, and run transactional reads. Change data feed must already be enabled on the table before it is shared with history for the recipient's change queries to return anything, so enabling it after the fact does not retroactively serve them. A table that uses deletion vectors or column mapping cannot be shared at all unless it is shared with history.

Trap Assuming a recipient can time travel or run an incremental stream against any shared Delta table because the provider's transaction log still holds the older versions, making the history option a storage-cost decision rather than a capability decision.

3 questions test this
Adding a whole schema to a share includes assets added to it later and always shares full history, but it forfeits table aliases and partition specifications

Schema-level granularity is the right choice when the consumer should keep receiving whatever lands in that schema, because every supported asset added afterwards joins the share automatically and everything travels with history. The cost is that the schema itself cannot be aliased and no table inside it can be given an alias or a partition specification. A requirement to expose one table under a different name, or to expose only certain partitions of it, therefore forces asset-by-asset sharing instead of the convenience of adding the schema.

Trap Expecting to add the whole schema for convenience and then alias or partition-restrict one table inside it, as if the two granularities could be mixed on the same shared object.

3 questions test this

Prepare and process data

Data Modeling in Unity Catalog

Read full chapter
  • Incremental extraction reprocesses only new files
  • Source file type dictates schema and evolution handling
  • Snapshot-only sources are diffed, not appended
  • Choose Lakeflow Connect for governed managed-source ingestion
  • Choose notebooks for custom or unsupported-source ingestion
  • Azure Data Factory is an orchestration or landing option
  • Pick streaming vs batch by latency SLA and arrival pattern
  • One streaming pipeline can run in batch or continuous mode
  • Delta is the default and required format for lakehouse features
  • CSV and JSON are ingestion or landing formats, not query targets
  • Iceberg reads (UniForm) expose one Delta copy to Iceberg readers
  • Do not partition small or high-cardinality tables
  • Liquid clustering replaces partitioning and Z-order
  • OPTIMIZE plus ZORDER BY compacts files and co-locates filter columns
  • Deletion vectors avoid rewriting files on delete or update
  • SCD Type 1 overwrites; Type 2 preserves history
  • Implement SCD with MERGE or AUTO CDC INTO
  • Choose table grain from the finest reporting requirement
  • Delta time travel queries bounded historical versions
  • Change Data Feed records row-level change events
  • SCD Type 2 provides durable temporal history
  • DROP deletes data for managed but not external tables
  • Only managed tables get automatic predictive optimization
  • External tables require an external location and credential
  • Unity Catalog managed Iceberg tables require serverless compute and predictive optimization
  • Managed Iceberg tables cannot carry Delta-only features such as constraints or generated columns
  • Deletion vectors and IcebergCompatV2 Iceberg reads cannot both be active on a Delta table
  • Iceberg clients see the last converted Delta version, not the newest Delta commit
  • Foreign Iceberg tables are read-only and time travel only to snapshots already read in Databricks
  • Identity values are unique and increasing but not contiguous, so a gapless-numbering or count-from-the-maximum requirement cannot be met with one
  • A surrogate key that must be reproducible across a full rebuild or identical in another environment has to be derived deterministically from the business key
  • An identity column can be neither added by ALTER TABLE nor defined by a CTAS, so retrofitting a surrogate key means recreating and reloading the dimension
  • A new lakehouse model should be a star or snowflake schema rather than a heavily normalized third-normal-form design
  • File-level statistics prune only for predicates against the table being read, so a selective filter that lives on a joined dimension can degrade into a full scan of the fact table
  • Collapsing every attribute into one wide table buys the skipping but pays it back as rewrite amplification and gives up the conformed dimension

Unlock with Premium — includes all practice exams and the complete study guide.

Ingesting Data into Unity Catalog

Read full chapter
  • Lakeflow Connect lands source data in Unity Catalog streaming tables
  • Lakeflow Connect supports batch and continuous sync
  • Notebook batch ingestion reads then appends to a table
  • Notebook streaming ingestion uses readStream with a checkpoint
  • COPY INTO is idempotent and loads only new files
  • CTAS creates once; CREATE OR REPLACE fully rebuilds
  • COPY INTO can infer types and evolve schema
  • Choose Auto Loader vs COPY INTO by file volume and cadence
  • AUTO CDC applies ordered CDC upserts and deletes
  • A CDC feed needs an ordering key and operation column
  • Checkpoints persist offsets for fast exactly-once resume
  • Trigger mode trades latency against cost
  • Each streaming query needs its own checkpoint location
  • Ingest Event Hubs via its Kafka-compatible endpoint
  • startingOffsets and maxOffsetsPerTrigger bound consumption
  • Auto Loader incrementally detects and loads new files
  • SDP streaming tables ingest incrementally; materialized views recompute
  • Auto Loader offers directory-listing or file-notification mode
  • Classic file notification mode builds one queue per stream and caps out per storage account
  • A stream that runs infrequently loses the file events fast path and falls back to listing
  • Classic file notification mode needs a backfill interval because event delivery is not guaranteed
  • Point file-events streams at a per-subpath Unity Catalog volume, not the external location root
  • Providing an explicit schema changes the default schema evolution mode to none
  • Schema inference and evolution happen only when cloudFiles.schemaLocation is set
  • Evolving the schema Auto Loader reads does not evolve the table it writes to
  • Schema evolution never picks up a new partition column inferred from the directory layout
  • Only the ingestion engine knows which files it has already committed, so an age-based storage lifecycle rule must not be aimed at the landing path
  • Archiving processed files keeps a recoverable copy but the archive must sit in the same container as the source, while deleting recovers nothing unless blob versioning is enabled
  • Expiring the stream's memory of processed files is a two-sided correctness risk, not a tuning win
  • A starting-position instruction is honoured only on a stream's very first start, so adding one to a stream that already owns a checkpoint changes nothing
  • A stream stopped longer than its source's retention resumes against positions the source has discarded, and skipping ahead buys the restart with a silent permanent gap
  • Records the event source has already discarded can be recovered only from a durable copy of the stream, never from the source itself

Unlock with Premium — includes all practice exams and the complete study guide.

Cleanse, Transform, and Load Data

Read full chapter
  • Profile data to get summary statistics and distributions
  • Distribution profiling exposes skew and outliers
  • Choose column types matching value domain and precision
  • Cast text source columns to typed columns on load
  • Parse and flatten semi-structured JSON into typed columns
  • Remove duplicates with DISTINCT or dropDuplicates on keys
  • Resolve nulls by dropping or imputing per requirement
  • NULL comparisons need null-safe operators
  • WHERE filters rows before grouping; HAVING filters groups
  • GROUP BY collapses rows into per-group aggregates
  • Join type controls which unmatched rows are kept
  • INTERSECT, EXCEPT, and UNION return distinct set results
  • Set operators require matching column count and types
  • Denormalize by joining dimensions into a wide table
  • PIVOT rotates row values into columns
  • UNPIVOT collapses columns into rows
  • MERGE performs upserts and SCD in one atomic statement
  • Append adds rows; overwrite replaces table or partition data
  • INSERT matches by position unless BY NAME is used
  • dropDuplicates on a stream retains every key as state until a watermark bounds it
  • dropDuplicatesWithinWatermark removes copies whose other fields differ
  • An insert-only MERGE does not remove duplicates inside the incoming batch
  • MERGE fails when more than one source row matches the same target row
  • Idempotent foreachBatch writes need a new txnAppId whenever the checkpoint is reset
  • REPLACE WHERE fails the write if any written row falls outside its predicate
  • REPLACE WHERE takes an arbitrary predicate and needs no matching partitioning
  • An empty source can delete rows under REPLACE WHERE but not under REPLACE USING or REPLACE ON
  • Dynamic partition overwrite replaces entire partitions and cannot combine with replaceWhere
  • A variant column stores an evolving payload whole and absorbs new or unknown keys without any change to the table
  • A variant column cannot be grouped, ordered, deduplicated, set-operated on or used as a partition or clustering key, so those fields must be shredded into typed columns
  • parse_json errors on a malformed or oversized payload while try_parse_json lands a NULL in its place
  • Declaring an identity column disables concurrent transactions on that table, so a target with more than one writer has to get its keys another way
  • GENERATED BY DEFAULT AS IDENTITY lets a reload carry the source system's existing keys in, while GENERATED ALWAYS refuses any supplied value
  • A generated column's expression must be deterministic and free of user-defined, aggregate and window functions, and declaring one upgrades the table's writer protocol

Unlock with Premium — includes all practice exams and the complete study guide.

Data Quality Constraints in Unity Catalog

Read full chapter
  • NOT NULL and CHECK constraints reject invalid writes
  • Use CHECK for range/cardinality; PK and UNIQUE are informational
  • The column schema enforces data-type validity
  • Delta schema enforcement rejects mismatched writes
  • Handle additive schema drift with schema evolution
  • Auto Loader preserves unmatched data in _rescued_data
  • Expectations offer warn, drop, and fail actions
  • expect_or_fail stops the update and rolls back
  • Dropped and failed record counts are tracked in the event log
  • expect_all applies multiple named constraints together
  • NOT ENFORCED and ENABLE NOVALIDATE attach to key constraints only, never to CHECK
  • A CHECK constraint is a per-row predicate and cannot look at other rows or tables
  • Dropping a check constraint does not undo the writer protocol upgrade it caused
  • Declaring an unverified key constraint RELY can produce incorrect query results
  • Automatic type widening needs schema evolution on the write AND the feature on the target
  • Setting enableTypeWidening to false removes neither the feature nor the applied changes
  • Widening is a closed list of numeric and date promotions; int to string is not one
  • Type widening raises the reader protocol and can break Iceberg reads
  • Anomaly detection is enabled on a schema and learns each table's own commit history to flag stale freshness and shortfalls in completeness, with no per-table configuration
  • A data profiling monitor is configured per table and is the only instrument that produces profile and drift metric tables
  • Data quality monitoring reports only after data has landed and never rejects a write or modifies the monitored table
  • Violating rows must be routed to a quarantine dataset because nothing in the pipeline retains a record it dropped
  • One non-discarding pass that flags validity on an intermediate streaming table can feed both the clean dataset and the quarantine dataset
  • Quarantining trades storing and governing bad data for keeping the correction-and-replay loop inside the pipeline

Unlock with Premium — includes all practice exams and the complete study guide.

Deploy and maintain data pipelines and workloads

Designing and Implementing Data Pipelines

Read full chapter
  • Pipeline order of operations follows the bronze-silver-gold medallion flow
  • Lakeflow SDP infers execution order from declared dataset dependencies
  • Separating ingest, transform, and serve stages lets each scale and recover independently
  • Choose SDP for declarative incremental ETL with built-in quality and orchestration
  • SDP manages infrastructure; a notebook task requires you to manage it imperatively
  • Notebooks suit procedural or non-ETL logic that cannot be expressed declaratively
  • A Lakeflow Job orchestrates a DAG of tasks linked by depends_on
  • Control-flow tasks add branching, iteration, and nested jobs
  • Pass data between tasks with task values and parameter references
  • Tasks come in several types beyond notebooks
  • A task retry policy automatically re-runs a failed task with a delay
  • Run-if conditions let downstream tasks run even when an upstream fails
  • Notebooks handle errors with try/except and surface status via dbutils.notebook.exit
  • %run inlines a notebook; dbutils.notebook.run executes it as a separate scope
  • Express precedence with Job tasks rather than deeply nested notebook calls
  • dbutils.notebook.run enables conditional branching and returns a string
  • SDP datasets are defined with pyspark.pipelines Python decorators
  • Streaming tables ingest incrementally; materialized views recompute results
  • Auto Loader is the recommended incremental file source inside SDP
  • SDP can also be authored in SQL with CREATE OR REFRESH statements
  • SDP development mode keeps compute warm; production mode restarts and retries
  • SDP pipelines update in triggered or continuous mode
  • A job that orchestrates a pipeline determines the execution mode and overrides the pipeline's own Pipeline mode setting
  • pipelines.trigger.interval controls update frequency only under continuous execution, never the cadence of a triggered pipeline
  • Setting the table property pipelines.reset.allowed to false stops a full refresh from ever clearing that one table
  • Tables excluded from a failed-table update are not refreshed even when they depend on a failed table
  • A repair run re-runs the unsuccessful tasks with the job's CURRENT settings, not the settings captured when the run started
  • Onboarding another source as one more append flow keeps an existing streaming table updating incrementally, while folding that source into the target's defining query rewrites its single flow and forces the target to be rebuilt
  • A history load that must happen exactly once is a ONCE flow, while a backfill location that keeps receiving late data needs an ordinary append flow left in place
  • The flow, not the streaming table, owns the streaming checkpoint, and a flow's identity is its name

Unlock with Premium — includes all practice exams and the complete study guide.

Implementing Lakeflow Jobs

Read full chapter
  • Each task chooses job compute, serverless, or an existing all-purpose cluster
  • Job parameters are key-value pairs passed to every task
  • Set Run as, max concurrent runs, and tags when configuring a job
  • Tasks can share one job cluster to cut startup cost
  • A notebook task reads job parameters with dbutils.widgets.get
  • Lakeflow Jobs supports scheduled, file-arrival, table-update, continuous, and manual triggers
  • Continuous keeps a job always-on; file arrival launches a run per batch of files
  • A table-update trigger runs a job when a monitored table changes
  • File-arrival triggers monitor a Unity Catalog external location or volume
  • A scheduled trigger uses a quartz cron expression with a timezone
  • Pause a schedule to stop runs without losing the job definition
  • Queueing controls what happens when a scheduled run overlaps a running one
  • Notifications attach at the job or the individual task level
  • Notifications go to email or admin-configured system destinations
  • Duration thresholds warn when a run is late or slow
  • Continuous jobs auto-restart the whole run with exponential backoff
  • SDP pipelines auto-recover; non-continuous jobs use per-task retries
  • Restart run cancels the active run and resets the backoff period
  • When a Run if condition is evaluated an Excluded upstream counts as successful, while Upstream failed or Upstream canceled counts as failed
  • Cancelling a task run propagates downstream and still runs the tasks whose Run if condition handles failure
  • Downstream tasks depend on the For each task, never on its nested task
  • A For each task cannot contain another For each task, and it fails outright when the upstream task supplying its inputs is disabled
  • A stage another team owns belongs in its own job invoked by a Run Job task rather than copied as tasks into every consumer job
  • Run Job chains may not be circular, directly or indirectly, and may not be nested arbitrarily deep
  • Whether a caller can trigger a child job and override its parameters is decided by the permission the caller's run-as identity holds on the CHILD job
  • A value an operator must vary per run belongs in a job parameter, because a static task parameter can only be changed by editing the task definition
  • A dynamic value reference is the layer that lets a task consume run metadata or another task's value at run time instead of a hard-coded literal
  • Only a task value can carry a value the job could not know before it started, and it flows forward along task dependencies only

Unlock with Premium — includes all practice exams and the complete study guide.

Development Lifecycle in Azure Databricks

Read full chapter
  • Databricks Git folders connect the workspace to a remote Git provider
  • Production jobs should reference a specific branch or tag, not a mutable copy
  • Never commit secrets to a Git folder
  • Branch, commit, and push from the Git folder UI; open PRs in the provider
  • Resolve merge conflicts by pulling main and reconciling text diffs
  • A merge to main can trigger CI that deploys the change
  • A layered strategy spans unit, integration, end-to-end, and UAT tests
  • Unit tests run with pytest; integration and E2E tests exercise real jobs on test data
  • A bundle can define a test job so CI runs tests on every deploy
  • A bundle is defined in databricks.yml with resources and targets
  • Development mode isolates deployments; production mode enforces production settings
  • Variables and target overrides parameterize a bundle per environment
  • databricks bundle validate checks the configuration before deploying
  • databricks bundle deploy -t promotes a bundle to a target environment
  • The REST API deploys resources programmatically using a service principal token
  • databricks bundle destroy removes deployed resources
  • The CLI takes a bundle variable's value from the first source that supplies one: --var, then BUNDLE_VAR_, then variable-overrides.json, then the target's variables mapping, then the default
  • A variable declared with a lookup resolves an existing object's ID from its name at deploy time, and errors when the name matches no object or more than one
  • Presets override a deployment mode's defaults, but an individual resource's own setting overrides the preset
  • Unattended bundle deployment authenticates as a machine identity through environment variables, and silently falls back to the DEFAULT profile when nothing is specified
  • Once run_as differs from the deploying identity, only jobs and pipelines are supported
  • Adopting an existing job or pipeline starts by generating its bundle definition from the deployed object and then moving the code it references into the bundle's own source tree
  • Binding a bundle resource to the existing workspace object is what makes the next deploy update that object instead of creating a second one, and the link is recorded per target
  • Unbinding is the deliberate release step that leaves the workspace object in place and no longer subject to later bundle operations
  • A permissions block declared at the top level of a bundle applies to every supported resource it defines, while a block on one job or pipeline covers only that resource
  • The access levels a specific resource type accepts are broader than the set a bundle-wide block allows, so a run-but-not-modify grant or pipeline ownership has to be declared on the resource itself
  • Operational access to bundle-deployed resources must be declared for a group rather than granted to individuals by hand in the workspace

Unlock with Premium — includes all practice exams and the complete study guide.

Monitoring, Troubleshooting, and Optimizing Workloads

Read full chapter
  • Auto termination stops an idle all-purpose cluster to remove idle cost
  • Autoscaling, pools, and spot instances trade cost against latency and interruption
  • System tables and tags attribute DBU consumption to workloads
  • Photon can improve price-performance despite a higher DBU rate
  • Repair run re-executes only the failed and dependent tasks
  • Repairing a non-idempotent task can duplicate its output
  • The run matrix view exposes per-task status and logs to locate a failure
  • Stop cancels an in-progress job run; Run now re-triggers or backfills
  • Use the Spark UI to decide whether to scale up or scale out
  • Driver out-of-memory from large collects may require a cluster restart
  • Verify Photon and an appropriate runtime version for performance work
  • Data skew creates stragglers that AQE or salting can relieve
  • Spill is data overflowing executor memory onto disk
  • Shuffle moves data across the network; broadcast joins avoid it for small tables
  • Disk cache and Spark cache serve different reuse patterns
  • The SQL query profile diagnoses slow SQL warehouse queries
  • OPTIMIZE compacts the many small files that frequent DML creates
  • VACUUM deletes unreferenced files beyond the retention window
  • Liquid clustering (CLUSTER BY) is the recommended alternative to partitioning plus ZORDER
  • Predictive optimization automates OPTIMIZE and VACUUM on managed tables
  • Diagnostic settings stream operational logs for troubleshooting
  • Azure Monitor alert rules fire actions on log or metric thresholds
  • The spark-monitoring GitHub library is legacy and in maintenance mode
  • By default the pipeline event log is a hidden table readable only by the pipeline's run-as user through the event_log() function
  • Expectation metrics are recorded for the warn and drop actions but not for fail, because the update never completes
  • Restarting a terminated compute replaces its Spark UI with the new instance, and an unpinned terminated compute is deleted after a bounded retention window
  • The compute event log records only that init scripts started and finished, so the failing script's output requires log delivery configured before the launch
  • Photon is always on for serverless compute, SQL warehouses and serverless pipelines, and is a per-compute choice only on classic all-purpose, jobs and pipeline compute
  • Photon earns its higher DBU rate only when rate multiplied by elapsed time falls, which happens on large scans, joins, aggregations and Delta writes but not on UDF, RDD or stateful streaming work
  • An operation Photon cannot run falls back to the Spark runtime transparently, so a bad fit announces itself only as unchanged runtime at a higher bill
  • A stream that is falling behind stays RUNNING with no failed task, so the verdict comes from the per-micro-batch progress report rather than from run state
  • A backlog proves the query is losing ground only when it grows monotonically, not when it is merely non-zero
  • The remedy for a lagging stream is a choice between bounding what each micro-batch admits, giving the query steadier compute, and moving it onto a runtime that scales the stream for you

Unlock with Premium — includes all practice exams and the complete study guide.