Domain 3 of 3 · Chapter 3 of 3

Optimize performance

Unlock the complete study guide.

Bundled into the existing Implementing Data Engineering Solutions Using Microsoft Fabric premium course — no separate purchase.

14-day money-back guarantee — no questions asked.

Included in this chapter:

  • Where Fabric performance work happens
  • Lakehouse Delta table maintenance
  • Controlling V-Order across Fabric engines
  • Tuning pipeline Copy activities
  • Warehouse query performance
  • Warehouse transactions and contention
  • Eventstream throughput
  • Speeding up Eventhouse queries
  • Spark compute tuning
  • Sizing Delta files at write time
  • Exam patterns and common traps

Lakehouse Delta maintenance: four operations, four jobs

QuestionOPTIMIZEZ-OrderV-OrderVACUUM
What it changesMerges many small Parquet files into fewer larger onesColocates related values of the chosen columns across filesParquet layout, encoding, and compression within filesDeletes files the Delta log no longer references
Run it whenSubstantial ingestion or updates left many small filesQueries filter selectively on specific columnsFiles are read repeatedly and the write cost is repaidStorage held by superseded files needs reclaiming
What it does not doDoes not delete unreferenced filesDoes not speed scans that ignore its columnsDoes not compact files or change writes already madeDoes not merge small files into larger ones
Default or boundary to rememberApplies to Delta tables, not Hive Parquet, ORC, AVRO, or CSVComplements V-Order and does not replace compactionFalse on new Spark workspaces, on by default for warehousesSeven-day retention threshold, and shorter is rejected by default

Decision tree

Is an external limit saturated?YesRaise the external limitSource, sink, or partition countNoWhich engine runs the work?Real-timeEventstream and EventhouseThroughput level, caching policy,query accelerationWarehouseWarehouse SQLStatistics, data types,early filtering, short transactionsPipeline or SparkMoving, computing, or reading files?MovingPipeline movementThroughput optimization,copy parallelism, ForEachComputingSpark computePool ceiling, repartition on skew,cache only reused dataReading filesLakehouse Delta table layoutOPTIMIZE, Z-Order, V-OrderVACUUM only reclaims storageAlways: read the evidence before and after the changeA first execution is never the baseline
Cheat sheet

Cheat sheet

  • OPTIMIZE compacts small Delta files for efficient reads
  • Z-Order optimizes selective filter access patterns
  • V-Order trades write throughput for repeated read efficiency
  • New Fabric Spark workspaces disable V-Order by default
  • Changing a V-Order table property affects only future writes
  • VACUUM removes unreferenced files subject to retention safety
  • Recurring lakehouse maintenance should be orchestrated and targets Delta tables
  • A session-level true setting forces V-Order for every write
  • Copy throughput is bounded by the slowest end-to-end resource
  • Auto throughput optimization adapts compute to the copy pattern
  • Degree of copy parallelism is a maximum thread count
  • Parallel copy and throughput optimization are independent controls
  • ForEach can parallelize independent copy operations
  • Source range partitioning enables parallel relational reads
  • ForEach parallelism applies across items, not within an item's child list
  • Tune a Copy run from measured stage and endpoint evidence
  • Accurate statistics guide Warehouse query plan selection
  • A cold first query execution is not a representative baseline
  • Matching comparison data types avoids conversion overhead
  • Early filtering and narrow projection reduce distributed query work
  • Warehouse V-Order defaults favor analytical reads
  • Short, closed transactions reduce Warehouse contention
  • Reducing single-node operations preserves MPP scalability
  • SQL pool pressure evidence separates resource problems from query-shape problems
  • Eventstream throughput tier should match sustained event volume
  • Insufficient source partitions can cap eventstream throughput
  • Eventhouse hot-cache duration balances query latency and cost
  • Query acceleration speeds Eventhouse queries over shortcut Delta data
  • Accelerated external tables retain external-table limitations
  • A materialized view precomputes a recurring KQL aggregation
  • Balance OneLake availability latency against Parquet file quality
  • Adaptive Query Execution optimizes shuffles and joins at runtime
  • Dynamic allocation scales Spark executors within pool bounds
  • Native Execution Engine accelerates supported Spark operations
  • Repartitioning corrects oversized or uneven Spark partitions
  • Spark caching should be reserved for reused data
  • High-concurrency sessions reuse Spark compute across notebooks
  • Tune Spark stream writes to produce sustainable Delta files

Unlock with Premium — includes all practice exams and the complete study guide.

Also tested in

References

  1. Monitor Apache Spark applications in Microsoft Fabric
  2. Copy activity performance and scalability guide
  3. Query insights in Fabric Data Warehouse
  4. Monitor the status and performance of an eventstream
  5. Performance guidelines for Fabric Data Warehouse
  6. Optimize Delta Lake tables with V-Order in Fabric
  7. Lakehouse table maintenance in Microsoft Fabric
  8. Cross-workload table maintenance and optimization
  9. Delta Lake table compaction in Microsoft Fabric
  10. Disable V-Order on a Fabric warehouse
  11. Configure Spark resource profile configurations
  12. ForEach activity in Fabric Data Factory pipelines
  13. Statistics in Fabric Data Warehouse
  14. Transactions in Fabric Data Warehouse
  15. Configure settings for a Fabric eventstream
  16. Kusto cache policy (hot and cold cache)
  17. Query acceleration over OneLake shortcuts
  18. Kusto materialized views overview
  19. Apache Spark compute for Fabric data engineering
  20. Native Execution Engine for Fabric Spark
  21. Apache Spark runtimes in Microsoft Fabric
  22. Autotune Apache Spark configurations in Fabric
  23. High concurrency mode in Fabric Spark
  24. Spark job concurrency and queueing in Fabric
  25. OneLake availability for Eventhouse (one logical copy)