Ingest and transform data

A retailer drops 400 store files into storage each night and wants them queryable as tables by 7 a.m.; a finance team wants only the invoices that changed since the last run; an operations team wants a live count of turnstile entries per minute. What separates those three requirements is not which Microsoft Fabric item you prefer, it is which of three questions each one asks: what shape the load has over time, how a bounded source lands and gets reshaped, and how a never-ending source is processed and queried. The shape question comes first because it is independent of the tooling, settling full against incremental copy, what the destination does with the rows that arrive, and what progress state carries between runs. The trap this model helps you dodge is an answer borrowed from a different question: a watermark, the stored value marking how far the last successful run got, answers "move less data each night" well and answers "reflect the rows the source deleted" wrongly, because watermark-based incremental copy sees inserts and updates but never deletions, which is what change data capture (a source database's own change feed) is for.

All three DP-700 domains carry the same 30 to 35 percent weight, and this one covers the movement itself: which rows arrive, in what shape, and which engine reshapes them. Design and implement loading patterns answers the shape question, pairing a copy mode (full or incremental) with an update method (append, overwrite, merge onto a key, or a new versioned row, which the table below lists under slowly changing dimension types) and the progress state that ties consecutive runs together, then applies the same thinking to dimension and fact loads. Ingest and transform batch data answers the bounded question, working through which store the data lands in, whether it arrives as a pointer, a replica, or a physical copy, and which engine shapes it once it is there. Ingest and transform streaming data answers the continuous question, choosing the streaming engine, deciding how Real-Time Intelligence (the Fabric workload for event and telemetry data) reaches the stored data, and picking the time window that turns an endless feed into a finite answer. The table and the router below ask those same three questions in the same order; the supported column types, published limits, and operator rules live on the three pages rather than here.

One instinct settles most ties in this domain: move as little as possible, then shape what is left in the engine that already owns the target workload. Moving less means preferring an incremental copy once a baseline exists, filtering at the source instead of after landing, and exposing data in place with a OneLake shortcut when the requirement never asks for an independent, governed copy. Shaping in place is possible because every Fabric store keeps its data in OneLake, the single tenant-wide data lake, so choosing a lakehouse, a warehouse, or an eventhouse, three of the five stores the table below names, is a choice of access experience and query language rather than of where the bytes physically live. The habit that rides along is to take the managed Fabric feature built for the job over a hand-built equivalent, since Copy job, not the Copy activity you place inside a pipeline, keeps its own incremental state and resumes from the end of the last successful run, and mirroring maintains a replica continuously, each removing a per-table pipeline you would otherwise write and make restartable.