Domain 3 of 3 · Chapter 2 of 3

Identify and resolve errors

Unlock the complete study guide.

Bundled into the existing Implementing Data Engineering Solutions Using Microsoft Fabric premium course — no separate purchase.

14-day money-back guarantee — no questions asked.

Included in this chapter:

  • Two layers of a Fabric failure
  • Pipeline failures: retry, condition, or fix
  • Copy fault tolerance and consistency checks
  • Dataflow Gen2 refresh diagnostics
  • Notebook and Spark job failures
  • Eventstream errors: decode, convert, or keep up
  • Eventhouse errors: capacity state before query
  • T-SQL errors: live requests, locks, and history
  • OneLake shortcut errors
  • Exam patterns and the traps they use

Engine-level diagnostic surface by failing item

Failing itemEngine-level diagnostic surfaceSignal that names the causeTime scope
Pipeline activityActivity output and error code in the run detailConnector error code, retry outcome, fault-tolerance skip logLive run plus Monitoring hub run history
Dataflow Gen2Recent runs, then downloaded Mashup engine logsException records, action paths, Duration against user and kernel CPULogs available minutes after completion, downloadable up to 28 days
Notebook or Spark jobContextual cell logs, Spark Advisor, Spark UIExecutor exits, skewed stages, storage and memory pressureLive during the run plus per-run application detail
EventhouseSystem overview and its database tilesRunning state, then query errors against ingestion errorsCurrent state
EventstreamData insights and runtime logsDeserialization against data conversion errors, backlog, watermark delayCurrent metrics plus logs filtered by severity and type
Warehouse or SQL analytics endpointLifecycle DMVs, then Query Insightsblocking_session_id, wait_type, wait_resource, long-running query shapesDMVs live now; Query Insights 30 days with up to 15 minutes lag
OneLake shortcutLakehouse shortcut settingsUnresolved target path, connection requiring action, permission intersectionCurrent state

Decision tree

Capacity state reported? Yes Capacity admin action increase or resume the capacity No Different on next attempt? Yes Retry, narrowed by condition error code, message, failure type No Needed only while it runs? Yes Live surface DMVs, contextual cell logs, eventstream data insights No Error code names the cause? Yes Fix what the code names credential, path, gateway, schema No Open the engine's own logs Mashup logs, Spark UI, runtime logs, Query Insights Always Read the engine surface before changing a setting
Cheat sheet

Cheat sheet

  • Activity retries are intended for transient failures
  • Conditional retries select failures by exposed error properties
  • The retry interval elapses before a retry condition is evaluated
  • Rerunning from the failed activity avoids repeating successful predecessors
  • Connector error codes should drive cause-specific remediation
  • Copy fault tolerance can skip incompatibilities without abandoning verification
  • Dataflow troubleshooting starts with the failed Recent run
  • Mashup engine logs provide deep Dataflow refresh diagnostics
  • A Dataflow refresh timeline must include every Mashup engine instance
  • Dataflow duration and CPU fields separate waiting from computation
  • Dataflow request and session identifiers correlate diagnostic records
  • Contextual notebook logs localize exceptions to the executing cell
  • Spark Advisor identifies known failure and skew patterns
  • Spark UI tabs isolate executor, stage, and storage root causes
  • Spark out-of-memory failures require memory-pressure correction
  • Backlogged input events signal an eventstream processing deficit
  • Conversion and deserialization metrics identify opposite schema boundaries
  • Eventstream runtime logs expose engine warnings and errors
  • Streaming connector metrics distinguish skipped records from failed processing
  • Eventhouse running state distinguishes service and capacity failures
  • Eventhouse overview separates query errors from ingestion errors
  • SQL lifecycle DMVs answer distinct live-diagnostic questions
  • Workspace role limits SQL diagnostic visibility and cancellation
  • Warehouse DML blocking is diagnosed as table-level locking
  • Query Insights diagnoses completed SQL executions over history
  • Moving or deleting a shortcut target can break the shortcut
  • Broken shortcut credentials are repaired by replacing the shared connection
  • Shortcut access is constrained by both shortcut and target permissions
  • Request wait columns identify both the blocker and the blocked resource

Unlock with Premium — includes all practice exams and the complete study guide.

References

  1. Monitor pipeline runs
  2. Activity overview
  3. General troubleshooting
  4. How to copy data using copy activity
  5. View Dataflow Gen2 refresh history and monitoring
  6. Understand Dataflow Gen2 detailed refresh logs
  7. Notebook contextual monitoring and debugging
  8. Apache Spark advisor for real-time advice on notebooks
  9. Spark errors overview in Microsoft Fabric
  10. Troubleshoot Spark memory and performance issues
  11. Monitor the status and performance of an eventstream
  12. Manage and monitor an eventhouse
  13. Monitor connections, sessions, and requests using DMVs
  14. Troubleshoot query blocking in Fabric Data Warehouse
  15. Query insights
  16. Unify data sources with OneLake shortcuts
  17. Manage shortcut connections
  18. Secure and manage OneLake shortcuts