DP-750 Cheat Sheet

Job (classic jobs) compute provisions a dedicated cluster scoped to a single job run and terminates it when the run finishes, isolating scheduled non-interactive pipelines from interactive development and billing at the lower jobs DBU rate.

Trap All-purpose compute is interactive and shared, so it is not recommended for production jobs and breaks the isolation requirement.

All-purpose compute is persistent, multi-user interactive compute for notebooks and ad hoc analysis; it bills at a higher DBU rate than job compute and is not recommended for production pipelines because it mixes development and production workloads.

A SQL warehouse is compute optimized for SQL queries and BI tools such as Power BI and Tableau, and comes in serverless, pro, and classic types; it cannot execute notebook or declarative-pipeline code.

Trap A SQL warehouse cannot serve as the compute engine for a Lakeflow declarative pipeline.

Serverless compute runs in the Databricks-managed cloud account and starts in seconds with no VM configuration or instance-pool management; it is available for notebooks, jobs, and declarative pipelines.

Cluster autoscaling automatically adds or removes worker nodes between a configured minimum and maximum based on load; it resizes only workers, never the driver, and it does not stop an idle cluster.

Trap Autoscaling only resizes an active cluster; it never terminates an idle one, which is the job of automatic termination.

A single-node cluster sets the worker count to 0 so the driver runs all Spark work with no worker nodes; it cannot autoscale and suits only small, non-distributed workloads.

Lakeflow Spark Declarative Pipelines run on runtime-managed job compute that uses enhanced autoscaling, which scales workers to workload volume and proactively shuts down under-utilized nodes to minimize cost.

Total cluster compute equals the number of worker nodes multiplied by each node's CPU and memory, so a workload is scaled either by adding workers (node count) or by selecting a larger node type.

A SQL warehouse t-shirt cluster size (X-Small to 4X-Large) sets the compute power for a single query, while its Scaling min/max range adds or removes WHOLE clusters to absorb concurrent-query load (not worker nodes, unlike cluster autoscaling); Auto Stop shuts an idle warehouse down (default 45 minutes classic/pro, 10 minutes serverless).

Trap SQL warehouse Scaling adds whole CLUSTERS for concurrency, whereas cluster autoscaling adds WORKER NODES; Auto Stop (warehouse) is distinct from a cluster automatic termination.

Automatic termination stops an all-purpose cluster after a set number of minutes with no Spark jobs, queries, or commands running, removing idle DBU and VM cost; a cluster is inactive only when nothing is executing, so active workloads keep it alive and are never interrupted.

Trap Enabling autoscaling or spot pricing lowers cost but does not stop an idle cluster; only automatic termination removes the idle waste.

An instance pool holds a set of idle, ready-to-use cloud VMs so clusters and autoscaling attach pre-acquired nodes and start faster; you pay the Azure VM cost for idle pooled instances but no DBU until a cluster uses them.

Configuring workers as Azure Spot VMs lowers per-hour compute cost, but Azure can reclaim spot capacity at any time, so spot fits fault-tolerant workloads and does not by itself reduce idle-cluster waste.

Photon is Databricks' native vectorized C++ query engine that transparently accelerates SQL and Spark DataFrame workloads, including JSON ETL and Delta operations; it is turned on per cluster with the Use Photon Acceleration checkbox and is on by default on recent runtimes.

Trap Photon accelerates SQL and DataFrame operations, not arbitrary Python or Scala UDFs.

Choosing a Databricks Runtime (DBR) version sets the bundled Apache Spark version and preinstalled libraries; Long Term Support (LTS) releases are recommended for production because they are supported and patched longer.

The Databricks Runtime for Machine Learning extends the standard runtime with preinstalled ML libraries such as PyTorch, TensorFlow, and scikit-learn (with GPU variants), so selecting it avoids manually installing the ML stack.

Cluster-scoped libraries install on the cluster and are available to every notebook attached to it, whereas notebook-scoped libraries installed with %pip apply only to the current notebook session and are discarded when the cluster restarts.

Compute libraries can be installed from package repositories (PyPI, Maven, CRAN) or from uploaded artifacts such as wheel, egg, or JAR files stored in workspace files or Unity Catalog volumes.

On standard (formerly shared) access mode compute, libraries and init scripts must be on the Unity Catalog allowlist maintained by a metastore admin, whereas dedicated access mode has no such restriction.

Trap The Unity Catalog allowlist governs standard access mode; dedicated (single-user) compute is not subject to it.

Compute access control assigns three permission levels to users, groups, or service principals: CAN ATTACH TO (run commands and view logs), CAN RESTART (start, stop, and restart), and CAN MANAGE (edit configuration and permissions).

A cluster's access mode determines sharing: standard access mode supports multiple concurrent users with full Unity Catalog governance, while dedicated access mode is assigned to a single user or group; both are Unity Catalog-enabled.

Trap The legacy No Isolation Shared access mode is not Unity Catalog-enabled.

A user can create all-purpose compute only if granted the unrestricted or policy-scoped cluster-creation entitlement; without it they can merely attach to existing compute they have been given access to.

A cluster (compute) policy is an admin-defined rule set that limits the compute a user may create or configure - allowed node types and Databricks Runtime versions, enforced auto-termination, maximum workers or DBU caps, and mandatory tags. Granting a user a policy scopes their cluster-creation entitlement to compute that complies with it, enforcing cost control and standards.

Trap A cluster policy governs the compute a user may CREATE (its configuration limits); that is different from a cluster ACL that grants Can Attach/Can Manage on an already-existing cluster.

Unity Catalog organizes every data object in a three-level namespace of catalog.schema.object (tables, views, volumes, functions, and models), replacing the legacy two-level hive_metastore.schema.table layout.

The catalog is the primary unit of data isolation in Unity Catalog, so a common naming convention creates a distinct catalog per environment (for example dev, test, and prod) to segregate data and permissions.

Trap Separating environments only by schema inside one shared catalog weakens the isolation boundary.

Workspace-catalog binding restricts a catalog so it is accessible only from designated workspaces, enforcing environment isolation and controlled external sharing across a metastore shared by several workspaces.

CREATE CATALOG creates the top-level container within the metastore; an optional MANAGED LOCATION sets where its managed tables and volumes store data, otherwise they inherit managed storage from the metastore.

Trap CREATE CATALOG builds a catalog, not a schema; a contained schema needs CREATE SCHEMA catalog.schema.

Only a metastore admin or a principal granted the CREATE CATALOG privilege on the metastore can create a catalog, and the creator becomes its owner with full control over the new object.

CREATE SCHEMA catalog.schema MANAGED LOCATION '' sets a schema's managed storage in Unity Catalog; the LOCATION clause is a Hive-metastore-only syntax and is rejected for Unity Catalog schemas.

Trap LOCATION is not supported for a Unity Catalog schema; use MANAGED LOCATION instead.

A schema (database) groups tables, views, and volumes and must be created inside a catalog with the two-level name catalog.schema; it cannot be created as a top-level object.

Setting a schema's MANAGED LOCATION requires the CREATE MANAGED STORAGE privilege on the external location that covers the path; without an explicit managed location the schema inherits managed storage from its catalog or the metastore.

A volume is a Unity Catalog object under a schema that governs access to non-tabular data such as images, CSV files, and model artifacts, accessed through the path /Volumes/catalog/schema/volume.

A managed volume stores its files in the schema's managed storage and is fully lifecycle-managed by Unity Catalog, whereas an external volume registers an existing path under an external location for data that Databricks does not own.

Trap Dropping a managed volume deletes its files; dropping an external volume leaves the files in place.

A view is a saved SELECT query that is evaluated each time it is read; it stores no data of its own and can restrict or reshape the columns exposed from its base tables.

A materialized view precomputes and stores query results and refreshes them incrementally through a Lakeflow declarative pipeline on serverless compute, speeding repeated reads at the cost of storage and scheduled refresh.

Trap A materialized view stores data and must be refreshed; a standard view does neither.

A Unity Catalog table stores rows and columns using the Delta Lake format by default, and can be created empty, with CREATE TABLE AS SELECT (CTAS), or with CREATE OR REPLACE TABLE.

To federate an external database you first create a connection object that stores the server host and credentials, then create a foreign catalog that uses that connection to mirror the external database's schemas in Unity Catalog.

A foreign catalog runs Lakehouse Federation queries directly against the source system so its schemas and tables appear alongside other Unity Catalog objects and are queried in place, with no data copied into Databricks-managed storage.

Trap An ingestion pipeline or a managed table would copy the data; only a foreign catalog queries the source in place.

A managed table stores its data files in Unity Catalog managed storage, so DROP TABLE removes both the table metadata and the underlying data files.

Trap Dropping an external table removes only the metadata and leaves the files intact.

An external table is created with a LOCATION clause pointing at a path under an external location, so Unity Catalog governs only its metadata and DROP TABLE leaves the underlying files untouched.

An AI/BI Genie space lets business users ask natural-language questions that Genie converts to SQL over a curated set of Unity Catalog tables, enabling self-service data discovery without writing queries.

Configuring Genie general instructions, example SQL queries, and trusted or certified answers guides how Genie interprets domain terms and business logic, improving the accuracy of the answers it generates for data discovery.

Trap Genie relies on the curated tables plus its instructions, not on unrestricted access to the whole metastore.

Access to a Unity Catalog securable is granted with GRANT ON TO , where the principal can be a user, an account group, or a service principal, and REVOKE removes it; common privileges include SELECT, MODIFY, and CREATE.

Trap A principal is not limited to individual users; groups and service principals are equally valid grant targets and groups are preferred for manageability.

To query a table a principal needs SELECT on the table plus USE CATALOG on its catalog and USE SCHEMA on its schema; the USE privileges grant traversal of the three-level namespace but do not by themselves expose any data.

Trap SELECT on the table alone is insufficient; without USE CATALOG and USE SCHEMA on the parents the query fails with a permission error.

Unity Catalog privileges are inherited down the object hierarchy, so a privilege granted on a catalog applies to all of its current and future schemas and tables, and a grant on a schema applies to its tables, views, and volumes.

In Unity Catalog the owner of a securable (a user, group, or service principal) implicitly holds all privileges on it and is the principal who can GRANT/REVOKE, ALTER, and DROP it; granting the MANAGE privilege lets a non-owner administer the object, including granting and revoking privileges and even dropping or transferring it, without being the owner, and ownership can be reassigned with ALTER OWNER TO.

Trap Holding SELECT is not enough to administer an object - only the owner or a MANAGE holder can grant, drop, or transfer it, and MANAGE differs from ownership only in that it is not automatically given the object's data privileges (it must self-grant SELECT).

Users, groups, and service principals in Unity Catalog are account-level identities, typically provisioned from Microsoft Entra ID via SCIM; account groups must be assigned to a workspace through identity federation before they can be granted privileges there.

Trap Groups are managed at the account level, not per-workspace - grants target account-level principals federated to the workspace, not a workspace-local group.

A SELECT grant applies to the entire table securable and cannot be scoped to individual columns, so column-level access control is achieved by layering a view that exposes only permitted columns, a column mask, or an ABAC policy on top of the base grant.

Trap Granting SELECT on only the non-sensitive columns is not a Unity Catalog capability; it would break SELECT * with a permission error rather than hide values.

A dynamic view wraps a base table and uses CASE expressions calling is_account_group_member() so that members of an authorized group receive the real column value while all other callers receive NULL or a redacted literal.

For view-based access control you grant SELECT on the dynamic or restricted view and do not grant access to the underlying base table, forcing every principal through the view where the column and row rules are enforced.

Row-level security through a dynamic view adds a WHERE predicate built from current_user() or is_account_group_member(), returning only the rows whose values match the querying principal, such as their own region or business unit.

A scalable row-level-security pattern joins the base table to a mapping table that records which principal or group may see which key values, so entitlement changes are made by editing data rather than by rewriting the view definition.

An Azure Key Vault-backed secret scope maps a Databricks secret scope onto an Azure Key Vault so notebooks can read its secrets; it is a read-only interface, meaning the secret values are created, updated, and rotated in Azure rather than in Databricks.

Trap A Key Vault-backed scope cannot be written from Databricks; secrets must be added and rotated in the Azure Key Vault itself.

Code retrieves a secret with dbutils.secrets.get(scope, key) instead of hardcoding credentials, and Databricks automatically replaces any printed secret value with [REDACTED] so it cannot leak into notebook cell output or logs.

Secret scope access control assigns per-scope ACLs at the READ, WRITE, and MANAGE levels, and a principal needs at least READ on the scope, which permits reading secret values and listing keys, before dbutils.secrets.get will succeed.

A service principal is an identity created for tools, jobs, and CI/CD rather than a person, and it is granted Unity Catalog privileges like any principal so automated pipelines can authenticate and access data without depending on an individual user's account.

Configuring a Lakeflow Job or pipeline to run as a service principal keeps it working when the original author leaves or loses access, and confines the job's data access to exactly the Unity Catalog privileges granted to that service principal.

A service principal authenticates non-interactively using OAuth machine-to-machine (M2M) tokens minted from its client ID and secret, which are the recommended automation credential in place of long-lived personal access tokens.

Resource access to ADLS Gen2 uses an Access Connector for Azure Databricks, a first-party Azure resource whose system- or user-assigned managed identity is granted a storage role such as Storage Blob Data Contributor, letting Databricks reach the storage account with no stored keys.

Trap A managed identity authenticates the underlying storage resource; a service principal instead represents a principal whose Unity Catalog privileges govern data access, not the storage connection.

In Unity Catalog the Access Connector's managed identity is registered as a storage credential, and an external location then references that credential to govern reads and writes to a specific ADLS Gen2 container path.

Authenticating storage access through an Access Connector managed identity is recommended over embedding storage account keys or SAS tokens, because the credential is managed by Azure and is never exposed in notebook code or cluster configuration.

Descriptions for discovery are set with the COMMENT clause or COMMENT ON and can be edited in Catalog Explorer; a table comment documents the dataset and per-column comments document each field, all stored as Unity Catalog metadata.

Catalog Explorer can suggest AI-generated table and column descriptions that a data steward reviews and accepts, accelerating the documentation of large catalogs so that objects become discoverable more quickly.

Because comments persist as Unity Catalog metadata they survive schema evolution and surface in Catalog Explorer search and AI/BI Genie, letting users find and understand data without opening the underlying files.

Governed tags are a centrally defined, account-level set of tag keys and allowed values, with permissions controlling who may apply each tag; they are the attributes that attribute-based access control policies evaluate to decide protection.

An attribute-based access control (ABAC) policy is attached at a catalog, schema, or table and uses governed-tag conditions to apply a row filter or column mask to every current and future object carrying the matching tag, so one policy governs many tables at once.

Trap ABAC scales a single tag-driven policy across many tables; a table-level SET MASK or SET ROW FILTER must be configured on each table individually.

A governed tag applied to a catalog or schema is inherited by the schemas and tables beneath it, but not by individual table columns, so tagging a parent propagates the attribute that ABAC policies and discovery rely on to descendant tables; a column-level classification must be tagged on the column directly.

A row filter is a boolean SQL UDF attached with ALTER TABLE SET ROW FILTER; the function is evaluated per row and returns TRUE to keep the row or FALSE/NULL to drop it, so it controls which entire rows a principal sees at query time.

Trap A row filter removes whole rows and cannot reveal only part of a column value; use a column mask for partial redaction.

A column mask is a SQL UDF attached with ALTER TABLE ALTER COLUMN SET MASK; it rewrites each returned value at query time, for example showing only the text after the @ in an email or only the last four digits of a card, while every row is still returned.

Trap Masking only sensitive columns while keeping table SELECT follows least privilege and lets queries run without errors, unlike revoking column access.

Table-level row filters and column masks are attached to base tables and cannot be placed on a standard view, while ABAC policies extend the same row-filter and column-mask protection to materialized views and streaming tables; once attached they are enforced for every query through any SQL warehouse, notebook, or client.

The table property delta.deletedFileRetentionDuration defines how long removed data files are retained before VACUUM is permitted to delete them, defaulting to 7 days; raising it lengthens the window during which older versions stay recoverable.

The table property delta.logRetentionDuration controls how long transaction-log history is kept, defaulting to 30 days, which bounds the versions and timestamps that time-travel queries can reference.

VACUUM permanently removes data files that are no longer referenced by the latest table state and are older than the retention threshold, after which you can no longer time-travel to a version whose files were purged, trading storage cost against recoverability.

For queries and workflows run on Azure Databricks, Unity Catalog captures runtime data lineage automatically down to the column level with no configuration, and aggregates it across every workspace attached to the metastore.

The Catalog Explorer Lineage tab renders an interactive graph of upstream and downstream tables, columns, notebooks, jobs, and dashboards alongside the object owner and history, so you can trace dependencies before changing or deleting an object.

The system.access.audit system table records account- and workspace-level audit events, capturing which principal accessed which securable and what action was taken, and retains them for up to one year for security and compliance analysis.

Configuring Azure diagnostic settings on the workspace streams Azure Databricks diagnostic (audit) logs to a Log Analytics workspace, a storage account, or Event Hubs for long-term retention, querying, and alerting in Azure Monitor.

Enabling verbose audit logging records additional fine-grained events such as commandSubmit, commandFinish, and runCommand, capturing notebook-cell and SQL-warehouse command activity that standard audit logging omits.

When the recipient also has a Unity Catalog-enabled workspace, a recipient of authentication type DATABRICKS shares data over a secure Databricks-managed channel identified by a sharing identifier, so no bearer token is created or managed and identity, authentication, and auditing are handled by the platform.

Trap Databricks-to-Databricks sharing eliminates token management; only open sharing to a non-Databricks recipient issues a bearer token.

Open sharing (the Databricks-to-open protocol) reaches recipients on any platform by authenticating a recipient of type TOKEN using a long-lived bearer token or OIDC federation; for a token, Databricks generates a credential file delivered via an activation link that must be secured and rotated.

A secure OpenSharing (previously Delta Sharing) strategy creates a share, adds only the specific tables or views to be exposed, and grants that share to a defined recipient, so the recipient receives read-only access to just the shared objects rather than to the whole metastore.