Monitoring, Troubleshooting, and Optimizing Workloads

• Auto termination stops an idle all-purpose cluster to remove idle cost
• Autoscaling, pools, and spot instances trade cost against latency and interruption
• System tables and tags attribute DBU consumption to workloads
• Photon can improve price-performance despite a higher DBU rate
• Repair run re-executes only the failed and dependent tasks
• Repairing a non-idempotent task can duplicate its output
• The run matrix view exposes per-task status and logs to locate a failure
• Stop cancels an in-progress job run; Run now re-triggers or backfills
• Use the Spark UI to decide whether to scale up or scale out
• Driver out-of-memory from large collects may require a cluster restart
• Verify Photon and an appropriate runtime version for performance work
• Data skew creates stragglers that AQE or salting can relieve
• Spill is data overflowing executor memory onto disk
• Shuffle moves data across the network; broadcast joins avoid it for small tables
• Disk cache and Spark cache serve different reuse patterns
• The SQL query profile diagnoses slow SQL warehouse queries
• OPTIMIZE compacts the many small files that frequent DML creates
• VACUUM deletes unreferenced files beyond the retention window
• Liquid clustering (CLUSTER BY) is the recommended alternative to partitioning plus ZORDER
• Predictive optimization automates OPTIMIZE and VACUUM on managed tables
• Diagnostic settings stream operational logs for troubleshooting
• Azure Monitor alert rules fire actions on log or metric thresholds
• The spark-monitoring GitHub library is legacy and in maintenance mode

Unlock with Premium — includes all practice exams and the complete study guide.