Monitoring, Troubleshooting, and Optimizing Workloads

• Auto termination stops an idle all-purpose cluster to remove idle cost
• Autoscaling, pools, and spot instances trade cost against latency and interruption
• System tables and tags attribute DBU consumption to workloads
• Photon can improve price-performance despite a higher DBU rate
• Repair run re-executes only the failed and dependent tasks
• Repairing a non-idempotent task can duplicate its output
• The run matrix view exposes per-task status and logs to locate a failure
• Stop cancels an in-progress job run; Run now re-triggers or backfills
• Use the Spark UI to decide whether to scale up or scale out
• Driver out-of-memory from large collects may require a cluster restart
• Verify Photon and an appropriate runtime version for performance work
• Data skew creates stragglers that AQE or salting can relieve
• Spill is data overflowing executor memory onto disk
• Shuffle moves data across the network; broadcast joins avoid it for small tables
• Disk cache and Spark cache serve different reuse patterns
• The SQL query profile diagnoses slow SQL warehouse queries
• OPTIMIZE compacts the many small files that frequent DML creates
• VACUUM deletes unreferenced files beyond the retention window
• Liquid clustering (CLUSTER BY) is the recommended alternative to partitioning plus ZORDER
• Predictive optimization automates OPTIMIZE and VACUUM on managed tables
• Diagnostic settings stream operational logs for troubleshooting
• Azure Monitor alert rules fire actions on log or metric thresholds
• The spark-monitoring GitHub library is legacy and in maintenance mode
• By default the pipeline event log is a hidden table readable only by the pipeline's run-as user through the event_log() function
• Expectation metrics are recorded for the warn and drop actions but not for fail, because the update never completes
• Restarting a terminated compute replaces its Spark UI with the new instance, and an unpinned terminated compute is deleted after a bounded retention window
• The compute event log records only that init scripts started and finished, so the failing script's output requires log delivery configured before the launch
• Photon is always on for serverless compute, SQL warehouses and serverless pipelines, and is a per-compute choice only on classic all-purpose, jobs and pipeline compute
• Photon earns its higher DBU rate only when rate multiplied by elapsed time falls, which happens on large scans, joins, aggregations and Delta writes but not on UDF, RDD or stateful streaming work
• An operation Photon cannot run falls back to the Spark runtime transparently, so a bad fit announces itself only as unchanged runtime at a higher bill
• A stream that is falling behind stays RUNNING with no failed task, so the verdict comes from the per-micro-batch progress report rather than from run state
• A backlog proves the query is losing ground only when it grows monotonically, not when it is merely non-zero
• The remedy for a lagging stream is a choice between bounding what each micro-batch admits, giving the query steadier compute, and moving it onto a runtime that scales the stream for you

Unlock with Premium — includes all practice exams and the complete study guide.