Domain 2 of 3 · Chapter 3 of 3

Ingest and transform streaming data

Unlock the complete study guide.

Bundled into the existing Implementing Data Engineering Solutions Using Microsoft Fabric premium course — no separate purchase.

14-day money-back guarantee — no questions asked.

Included in this chapter:

  • Where this page sits in the streaming path
  • Choosing a streaming engine
  • Bringing events into an eventstream
  • Eventstream limits that decide feasibility
  • Transforming events in the event processing editor
  • Destinations and the derived stream bridge
  • The SQL operator for code-first stream logic
  • Windowing functions and event time
  • Governing evolving event schemas
  • DeltaFlow for analytics-ready CDC streams
  • Spark Structured Streaming into a lakehouse
  • Native tables against OneLake shortcuts
  • Query acceleration for OneLake shortcuts
  • OneLake availability for KQL tables
  • Aggregating events with KQL
  • Exam-pattern recognition

Reaching stream data in Real-Time Intelligence

CriterionNative Eventhouse tableStandard OneLake shortcutAccelerated OneLake shortcut
Data copyIngested copy stored in the EventhouseNo copy; data stays at the sourceNo second source copy; a cached hot period is held by the Eventhouse
How it is queriedBy table name in KQLexternal_table('<shortcut-name>')external_table('<shortcut-name>')
IndexingOrganized and indexed on ingestionNo indexes; queries fetch from source storageCached and indexed for the configured hot period
Materialized views and update policiesSupportedNot supportedNot supported
Source schema changesHandled by the ingestion mappingColumn adds, deletes, renames, and type changes sync by default; disable with AutoUpdateSchema=falseSynced the same way, but a breaking schema change can restart acceleration from scratch
Shape limits to check firstStandard KQL table policies applyWhatever the source layout imposesAt most 900 columns; Parquet files over 1 GB are not cached (policy reference); over 2.5 million data files may not be optimal
Main cost driverEventhouse storage plus ingestion computeQuery-time reads against the sourceOneLake Premium cache meter plus indexing compute

Decision tree

Supported eventstream connector or Kafka protocol source? No Custom endpoint source your producer posts to the eventstream endpoint Yes Needs custom code or state across micro-batches? Yes Spark Structured Streaming you own the code, checkpoint, and the running job No Interactive KQL over high-volume events? Yes Eventstream to an Eventhouse native table, indexed on ingestion for KQL No Eventstream to a lakehouse Delta table for Spark and SQL endpoint reads
Cheat sheet

Cheat sheet

  • Choose Eventstreams for connector-driven no-code stream processing
  • Choose Structured Streaming for code-first distributed stream logic
  • Choose Eventhouse when rapid KQL analysis is the primary outcome
  • Choose a lakehouse sink for durable Delta data shared with Spark
  • Balance latency, transformation complexity, and operations when selecting an engine
  • Use a native Eventhouse table for fully indexed ingestion performance
  • Use a KQL shortcut to query OneLake data without moving it
  • Do not assume external shortcut tables support native table features
  • Account for automatic schema synchronization on shortcut tables
  • Enable OneLake availability to expose native KQL data as Delta
  • Account for KQL table restrictions while OneLake availability is enabled
  • Enable query acceleration for frequently queried external Delta data
  • Set the accelerated hot window from Delta modification time
  • Budget storage and compute for accelerated shortcuts
  • Check external table shape before enabling acceleration
  • Disable acceleration before changing advanced Delta table features
  • Provide an upstream schema before configuring visual operators
  • Insert a derived stream when a destination cannot accept an operator directly
  • Enable enhanced capabilities when transformations must feed all destinations
  • Keep one SQL operator as the transformation boundary for a path
  • Govern evolving Eventstream schemas with registry-aware paths
  • Use DeltaFlow to normalize supported CDC streams for analytics
  • Write append-only Spark streams to Delta with a separate checkpoint
  • Use KQL summarize with time bins for interval aggregates
  • Use TIMESTAMP BY when windows must follow event time
  • Use tumbling windows for fixed nonoverlapping intervals
  • Use hopping windows for overlapping rolling evaluations

Unlock with Premium — includes all practice exams and the complete study guide.

References

  1. Microsoft Fabric eventstreams overview
  2. Data streaming into a lakehouse with Spark
  3. Eventhouse overview
  4. Create OneLake shortcuts in a KQL database
  5. Query acceleration for OneLake shortcuts overview
  6. Stateful processing with Structured Streaming
  7. Turn on OneLake availability for an eventhouse
  8. Add a custom endpoint or custom app source to an eventstream
  9. Process event data by using the event processing editor
  10. Add and manage eventstream destinations
  11. Add a derived stream destination to an eventstream
  12. Azure Stream Analytics and Fabric Eventstream query language reference
  13. Process events using a SQL operator
  14. Windowing - Stream Analytics query
  15. Tumbling window - Stream Analytics query
  16. Hopping window - Stream Analytics query
  17. Use schemas in eventstreams
  18. Enhance event processing with multiple schema inferencing
  19. DeltaFlow output transformation (Preview)
  20. Apache Spark job definition
  21. Apache Spark monitoring overview
  22. external_table() function
  23. https://learn.microsoft.com/en-us/kusto/management/query-acceleration-policy
  24. summarize operator
  25. Create and edit materialized views
  26. Create a table update policy in Real-Time Intelligence