Domain 1 of 3 · Chapter 4 of 4

Orchestrate processes

Choosing between Dataflow Gen2, a pipeline, and a notebook

Sales files land in a storage container every night, need deduplicating and conforming before anyone can query them, and a semantic model (the tabular data model that Power BI reports query) has to refresh once the load is clean. Three Fabric items can carry parts of that job, and choosing the wrong one is the most common way an orchestration design goes wrong before a single activity ever runs. Building that nightly job end to end takes four decisions, and making them is what this page equips you to do: which item does the work, what starts the run, how values reach it, and what happens on each step's success and failure paths.

A neighboring page answers a different question. Ingest and transform batch data weighs the transformation engines against each other: Dataflow Gen2, notebooks, KQL, and T-SQL. This page is about the items that run and coordinate the work, how a run gets started, and how the steps are sequenced once it does.

The three items

A data pipeline is a workflow made of activities, where an activity is one configured step on the pipeline canvas. Fabric sorts activities into three families: data movement, data transformation, and control flow. A pipeline can hold up to 120 activities by default, and that count includes activities nested inside containers such as ForEach (Activity overview[1]).

Dataflow Gen2 is the low-code transformation item, authored in Power Query[2]. It is the successor to the original Power BI dataflow, which Fabric now calls Dataflow Gen1, so a doc or a colleague saying just "dataflow" may mean either; this page always means Gen2. It reaches 150+ connectors, offers 300+ transformation functions, and writes to a fixed list of output destinations that currently covers Fabric Lakehouse tables and files, Fabric Warehouse, Fabric SQL database, Fabric KQL database, Azure SQL databases, Azure Data Explorer, Azure Data Lake Gen2, SharePoint files, Snowflake, and PostgreSQL (Differences between Dataflow Gen1 and Dataflow Gen2[3]). Since April 2026, every new Dataflow Gen2 item is created with CI/CD and Git integration support by default; existing items created without it keep working.

A Fabric notebook is the code-first item. It runs Python, Scala, SQL, or R on Apache Spark clusters managed by Fabric, which is why its transformation ceiling is set by the Spark and open-source libraries you can import rather than by a fixed function list.

Reading the choice off the work

The selection criteria come from the work, not from preference, and Microsoft publishes them as a property comparison (Fabric decision guide[4]).

When the transformation is standard reshaping that low-code users have to keep maintaining, Dataflow Gen2 is the fit. Dropping such a transformation into a notebook trades away that maintainability for Spark capability the job does not need, which is a real cost paid for no benefit.

When the work needs custom Python, Scala, SQL, or R, or needs distributed processing for very large files or a custom algorithm, use a notebook. This is also where transformations go once they exceed what Power Query can practically express.

When the requirement is to coordinate several of those pieces, with ordering, conditional paths, and retries, only a pipeline does that. Dataflow Gen2 transforms data and cannot orchestrate other items, so an answer that nominates it as the coordinator of several child pipelines is wrong on capability, not on style.

These are not exclusive choices, and in practice they are usually combined. A pipeline runs a Dataflow Gen2 through the Dataflow activity[5] and a notebook through the Notebook activity[6], so the common production shape is one pipeline conducting both. The rest of this page follows that shape: what starts a run, how values move through it, and how the steps are wired together.

Scheduling pipeline runs

Fabric pipeline schedules are bounded, never open ended. Every schedule carries a start date, an end date, a frequency, and a time zone (Run, schedule, or use events to trigger a pipeline[7]). There is no option for an open-ended schedule, so the documented way to keep a pipeline running long term is to set the end date far in the future, for example 01/01/2099 12:00 AM, and update or stop the schedule when the requirement changes. Leaving the end date blank is not an available configuration, and an answer that offers it is a distractor.

A single pipeline can carry up to 20 schedules, added one at a time, each with its own frequency and its own start and end times.

Fixed and interval-based schedules

There are two schedule types, and picking between them comes before configuring either one.

A fixed schedule runs at the times its frequency describes. It is the ordinary choice, and it is the only type you can enable, disable, or edit in place after saving.

An interval-based schedule, currently in public preview, configures fixed, non-overlapping intervals instead. Its distinguishing feature is that it surfaces two extra values in the pipeline expression builder under Trigger parameters, named Window start time and Window end time, so each run can bound its own query or file selection to the slice of time that run owns. Two limits travel with it. Time slice monitoring and backfill are not yet available, so missed windows are not replayed automatically. And an interval-based schedule cannot be enabled, disabled, or edited after it is saved: changing one means deleting it and creating a replacement.

Schedule types compared

Property Fixed schedule Interval-based schedule (preview)
Window boundaries in expressions Not provided Window start time, Window end time
Edit, enable, or disable after saving Yes No, delete and re-create
Automatic backfill of missed windows Not applicable Not available

Passing parameter values from a schedule

A schedule can supply values for the pipeline's parameters at each execution, in one of two forms. A Direct value is a static value typed into the schedule configuration. A Variable library reference pulls the value from a Variable library item instead, which keeps configuration central and simplifies promotion between environments.

The parameter names configured in the schedule must exactly match the names defined in the pipeline. Mismatched names are ignored at run time rather than raising an error, so a typo here produces a silently wrong run instead of a failed one.

Failure notifications and who may configure them

Email notifications for failed scheduled runs are set up by adding users or groups under Failure notifications in the schedule pane. Three boundaries matter more than the setup itself (Job Scheduler in Microsoft Fabric[8]).

Notifications are sent only for failures from scheduled runs, and are not sent for manually triggered runs. A manual test therefore produces no email regardless of who is listed.

Notifications are configured at the item level, and the settings apply across all schedules for that item rather than to the one schedule open in front of you. Recipients can be users or groups in the Microsoft Entra tenant, including B2B guest users; direct external email addresses are not supported.

To set up or modify a schedule, or to configure failure notifications, you need at least the Contributor role in the workspace or Write permission on the item. Users with only Viewer access can see existing schedules but cannot change them.

Two platform behaviors sit behind the scheduler and explain runs that stop without anyone touching them. A scheduler that hits consecutive failures is shut down into an auto-disabled state and must be restarted manually through the UI, API, or CI/CD process; the threshold varies by item and is typically 10 consecutive failed runs. Separately, schedules expire if a user does not sign in to Fabric for 90 consecutive days.

The through-line for this section: a schedule is a bounded, item-level object, not a property of a single run. Its time window, its parameter bindings, its notification recipients, and the permission needed to touch any of them all live with the item, which is why so many of its behaviors surprise people who expect per-run settings.

Event-based triggers and Activator rules

A requirement worded "start ingestion when the file arrives" is asking for an event trigger, and no schedule frequency satisfies it. A one-minute schedule polls rather than reacts, and it spends capacity on the hundreds of runs per day that find nothing. The distinction is what the exam tests here, so it is worth stating plainly: a schedule asks when, a trigger asks what happened.

Event triggers start a pipeline when file events, job events, or workspace events occur. They are built on two platform features rather than on the pipeline itself. Fabric eventstreams carry the events, and Activator alerts (written Data Activator in some docs; the same feature) evaluate the condition and take the action.

What the setup actually creates

Starting from the Trigger button on the pipeline's Home ribbon, the setup walks through the event source (OneLake events or Azure Blob Storage events, among the sources offered), the event types to listen for, and a Subject filter that narrows to a folder name, file name, file type, and container. Two objects come out of it. A new eventstream object is created in the workspace, and the trigger itself is saved as a Reflex item, which is the object type to look for in the workspace list when you later need to view, edit, or stop the trigger.

Activator rules start more than pipelines

The same Activator machinery reaches further than the pipeline you set it up from. An Activator rule can take an action that starts a job on a Fabric item, and the supported targets are Fabric pipelines, Dataflows, notebooks, Spark job definitions (a Spark application submitted as a batch job rather than run interactively), User Data Functions (custom business logic packaged as a callable function), Copy jobs (the wizard-driven data movement item), and Publish business event, which is in preview (Trigger Fabric items[9]). The action is configured in the Action section of the rule definition, and the target item is picked from the OneLake catalog, the workspace-wide item browser.

Passing parameter values to the activated item is in preview and carries one strict requirement: define the parameter name and type exactly as they are defined in the target Fabric item. Values are entered manually or selected as dynamic properties from the data source. Activator accepts string, Boolean, and number (float) parameters, and supports every parameter type that User Data Functions themselves support. One target does not participate at all, and it is a recurring exam detail: Copy jobs don't accept parameters.

So an event-driven design is really two decisions: which event should wake the work, and which item should do it. The eventstream and the Reflex item are plumbing that the setup creates for you; the Activator rule is the piece you actually author.

The figure below traces the storage-event chain from the event through to the pipeline run and the parameters that run receives.

Storage event BlobCreated Eventstream created in workspace Activator rule condition and action Reflex item holds the trigger Pipeline run starts Trigger parameters FileName, FolderPath available to expressions
How a storage event reaches a pipeline run in Fabric: eventstream, Activator rule, Reflex item, and the trigger parameters exposed to expressions.

Parameters, variables, and dynamic expressions

Two constructs hold values inside a pipeline and they are not interchangeable: a parameter's value is fixed for the whole run, while a variable exists precisely so that it can change during one.

Parameters pass external values into a pipeline. You define them on the Parameters tab of the pipeline editor canvas with a name, a data type, and a default value, and after a run starts, the value stays the same throughout that run (Parameters[10]). That immutability is what makes one pipeline reusable across many sources: the same activities, a different set of parameter values on each run.

Variables are the construct for state that has to move. A counter incremented inside a ForEach loop, a list built up with Append variable, a flag flipped by Set variable: each of those is a variable. Reaching for a parameter as a counter is a design error the run will not report as such, because the assignment simply has nowhere to land.

One naming collision is worth settling here, because four different things on this page are called parameters. Pipeline parameters are the ones defined on the pipeline itself, described above. Trigger parameters are supplied by the platform from a schedule window or a storage event. Base parameters are what a Notebook activity discovers from a notebook and passes into it. Dataflow parameters are the ones a Dataflow Gen2 exposes to its calling activity. They share a word and nothing else: different owner, different lifetime, different place to configure them.

Referencing values, and the type the reference returns

An expression can appear inside a string through string interpolation, where the expression is wrapped in @{ ... }. This is where a small syntax choice changes the data type an activity receives.

Expression forms and their return types, for myNumber = 42 and myString = foo

Expression value Result
@pipeline().parameters.myString Returns foo as a string
@{pipeline().parameters.myString} Returns foo as a string
@pipeline().parameters.myNumber Returns 42 as a number
@{pipeline().parameters.myNumber} Returns 42 as a string
Answer is: @{pipeline().parameters.myNumber} Returns the string Answer is: 42

The rule those rows encode is short. String interpolation always returns a string; a direct expression reference preserves the native type. So when an activity property requires a number, a Boolean, an array, or an object, reference the expression directly. Wrapping it in @{...} to be safe does the opposite of safe, because the property then receives text.

Two neighboring pieces of the same syntax are worth pinning down at first touch. To use a literal string that begins with @, escape it by typing @@, so "@@" yields a one-character string containing @. And inside a ForEach, @item() returns the current element of the collection being iterated: for the items [1, 2, 3] it returns 1 on the first iteration, 2 on the second, and 3 on the third (Expression language[11]).

System variables that describe the run itself

A set of pipeline-scope system variables can be referenced anywhere in the pipeline and answer questions about the current execution rather than about your data. @pipeline().RunId is the ID of this specific run, @pipeline().Pipeline its name, @pipeline().TriggerName and @pipeline().TriggerTime identify the trigger that invoked it and when it actually fired, and @pipeline()?.TriggeredByPipelineName names the calling pipeline when another pipeline started this run through an Invoke pipeline activity, evaluating to Null in every other case.

The ? in that last one is the same null-safe operator the trigger parameters need, for the same reason: the field is legitimately absent in some runs. Trigger-related date and time system variables, in both pipeline and trigger scopes, return Coordinated Universal Time in ISO 8601 format, for example 2017-06-01T22:20:00.4061448Z, so a comparison against a local-time value has to convert first.

Where a manual test run differs

The built-in trigger parameters from an event-based start deserve one more line here, because they interact with parameters and expressions directly. Data Factory parses the file name and folder path out of the storage event's Subject and Topic fields and exposes them in the expression builder, referenced as @pipeline()?.TriggerEvent?.FileName. During a manual test run there is no storage event, so those parameters are unset and return NULL. The null-safe ? syntax is required rather than stylistic: without it, an on-demand test fails on the expression rather than on the logic you meant to exercise.

Two habits carry this section: decide whether a value is fixed for the run (parameter) or changing during it (variable) before you write the expression, and check what type the reference returns before you hand it to an activity property.

Dependency conditions between activities

The arrow between two activities is not just sequence, it is a condition, and there are exactly four of them.

On success runs the downstream activity only when the upstream one succeeded. On failure runs it only when the upstream one failed. On completion runs it after the upstream activity either way. On skip runs it only when the upstream activity was skipped, which happens when that activity's own incoming condition was not met. Reading a pipeline means reading which of the four each arrow carries: two activities can sit adjacent on the canvas and never both run.

The figure below shows one activity's four outgoing paths.

Upstream activity On success upstream succeeded On failure upstream failed On completion either outcome On skip upstream skipped
The four activity dependency conditions leading out of one upstream activity in a Fabric data pipeline.

Control flow activities

Branching and looping are activities in their own right, and picking among them is a question about the shape of the decision rather than about the data (Activity overview[1]).

If Condition branches on a condition that evaluates to true or false, running one set of activities for each result. Switch implements a switch expression, allowing multiple subsequent activities for each potential result of one expression, which is the right choice when there are more than two outcomes selected by a single value. Until implements a do-until loop, executing a set of activities repeatedly until the condition associated with it evaluates to true, and it accepts a timeout value. ForEach iterates over a collection and runs its child activities once per item. Filter applies a filter expression to an input array, which is how a collection is narrowed before a ForEach rather than by branching inside the loop.

A deactivated activity confirms that the condition, not the execution, chooses the path. A deactivated activity never actually runs, but you pick what to Mark activity as, and the branch follows that choice: marked Succeeded runs the On success or On completion branch, marked Failed runs On failure or On completion, and marked Skipped runs On skip. That also makes deactivation a clean way to comment out part of a pipeline during development without deleting it from the canvas.

ForEach parallelism, stated without ellipsis

ForEach concurrency is item-level, and the compressed version of this rule is where the mistakes come from. On the ForEach Settings tab, Sequential processes the items one at a time. Leave it unselected and Batch count caps how many items are processed at the same time. Either way, the child activities inside the ForEach are processed sequentially with respect to one another for each item (ForEach Activity[12]).

So clearing Sequential parallelizes items; it never parallelizes the steps within one item. Multiple items are processed in parallel, each of them running sequentially through the list of child activities. An answer that clears Sequential in order to make one item's child activities run simultaneously is wrong about which axis the setting controls.

Letting the data choose the path

Two activities exist to turn data into control flow instead of hard-coding it. Lookup reads or looks up a record, a table name, or a value from an external source, and its output can be referenced by succeeding activities. Get metadata retrieves metadata of data in a pipeline, such as whether an item exists, its structure, or its column count.

The pattern they compose into is the workhorse of parameterized orchestration: Lookup a configuration table, feed its rows into the ForEach Items property, reference the current row with @item() inside the child activities, and branch on a Get metadata result rather than encoding the list of sources into the pipeline. Adding a source then means adding a row, not editing a pipeline.

Per-activity failure and logging settings

This section covers the per-activity settings on an activity's General tab that decide how a step behaves when it fails and what it writes into the run logs. They are configured per activity, so each one is a local decision rather than a pipeline-wide policy.

Retries

Most pipeline activities support automated retries, so when one fails on a transient error such as a throttled API, a brief service outage, or a flapping dependency, retry settings handle it with no orchestration you have to build yourself. Select Enable retries, then set Retry to the number of attempts, a value between 1 and 1000 with a default of 1 (Activity retries[13]). Wrapping a Copy activity in a ForEach loop to emulate retries adds a construct the platform already provides, and it loses the back-off behavior described next.

Retry interval type chooses how the wait between attempts is calculated:

Type Behavior Defaults
Fixed (default) Waits the same number of seconds between every attempt Retry interval 30 seconds
Increasing Delay Exponential back-off: each retry waits a random interval drawn from a range that grows exponentially, up to a configurable maximum Retry interval 30 seconds, Max retry interval 3600 seconds

The range doubles with each attempt until its upper bound reaches Max retry interval, after which every remaining retry waits at that maximum. Drawing randomly inside each range is deliberate: it reduces the chance that concurrent pipeline retries collide on the same upstream system at the same moment.

By default an activity retries on any failure. Retry conditions, in preview, narrow that to specific errors so attempts are not spent on failures that will never resolve, such as an authentication error. A condition matches on one of three fields: Error message (the text content of the error), Failure type (the category, for example User error or System error), or Error code (the specific code returned, for example 429 for rate limiting). Conditions combine with And or Or. Conditional retries are supported for specific activity types, including Copy data, Notebook, Dataflow, and Stored procedure activities, so this is not a setting you will find on every activity.

One ordering detail catches people out: the retry interval runs before the condition is evaluated. With a one-hour retry interval and a condition that turns out not to match, the pipeline still waits the full hour before proceeding to the next activity or ending the run. Retry interval fields also do not accept dynamic expressions; their values must be static integers.

Timeouts

Timeout controls how long an activity may run before it is cut off. The default is 12 hours, the maximum is seven days, and the value uses the format D.HH:MM:SS.

One scope qualifier belongs with that number rather than after it. For the Dataflow activity, timeout and cancellation apply only to Dataflow Gen2 with CI/CD support. For runs of a Dataflow Gen2 without CI/CD support, timeouts are ignored and cancelling the pipeline does not stop the dataflow run.

Keeping values out of the logs

Two settings under Advanced properties control what run logging records. Secure input, when selected, keeps the activity's input from appearing in logs. Secure output does the same for its output.

Apply them to the activities whose parameters, credentials, or returned values would otherwise be readable by anyone with access to the run history. Because both are per-activity settings, securing one activity in a chain does not secure the activity that consumed its output: a secret passed from a secured Web activity into an unsecured Notebook activity is back in the logs at the second step.

Running notebooks from pipelines

A Notebook activity is how Spark code becomes one step of an orchestrated workflow. The pipeline supplies the scheduling, the dependency conditions, the retries, and the parameter values, and the notebook keeps doing the code-first processing it is good at (Notebook activity[6]).

Parameters come from the parameter cell, and only from there

Add the activity, choose an authentication method under Connection, then select the notebook from the Notebook dropdown. Auto-populate base parameters discovers the notebook's parameters and fills them into the activity during authoring, which keeps the pipeline aligned with the notebook's own logic instead of with a hand-maintained copy of it.

The discovery has one prerequisite that governs the whole behavior: the notebook must contain a parameter cell, and only variables defined in that cell are detected as parameters. Variables declared in ordinary cells are invisible to the activity no matter how they are written. When you select a notebook, the Base parameters section populates with names, types, and default values, which you can review and optionally override with different values for this pipeline. Selecting a different notebook refreshes the list, and a notebook with no parameter cell populates nothing at all.

Session tags need a workspace setting to do anything

Set a session tag on a Notebook activity to cut the startup time of a notebook job. The tag instructs Spark to reuse an existing session rather than creating a new one, any arbitrary string works as the value, and if no session exists with that tag, a new one is created using it.

The setting only takes effect if the workspace permits it. High concurrency mode for pipeline running multiple notebooks, found under the High concurrency mode Spark settings in Workspace settings, must be turned on. Identical session tags with that option left off produce no session reuse whatsoever, and nothing reports a problem: the notebooks simply each pay full Spark startup time. Workspace-level Spark configuration is covered on Configure Microsoft Fabric workspace settings.

One boundary belongs with the activity itself: the Notebook activity does not support calling notebooks across regions.

Notebook-driven orchestration with NotebookUtils

Not every chain of notebooks needs a pipeline wrapped around it. NotebookUtils is a built-in package for Fabric notebooks, available in PySpark, Scala, and SparkR notebooks and in Fabric pipelines, and designed for Spark 3.4 (Runtime v1.2) and above (NotebookUtils for Fabric[14]). It was renamed from MSSparkUtils; existing mssparkutils code remains backward compatible and does not break, but that namespace is slated for retirement, so new work uses notebookutils.

The module that matters for orchestration is notebookutils.notebook, which runs and chains notebooks together, including parallel execution with DAG (directed acyclic graph) support, cross-workspace references, and exit values that let one notebook return a result to whichever notebook called it. Running notebookutils.notebook.help() lists its methods from inside a notebook.

Choosing between the two orchestration surfaces follows the same logic as choosing the items themselves. A pipeline is the right home when the workflow spans different item types, needs conditional paths, or needs schedules and event triggers, none of which NotebookUtils provides. NotebookUtils is the right home when the whole workflow is notebooks and the person maintaining it works in code.

Invoke pipeline: parent and child pipelines

A pipeline that does everything on one canvas becomes unreadable and impossible to reuse. The Invoke pipeline activity is what lets a workflow be built instead as a small parent that sequences purpose-built children.

Two Invoke pipeline activities, and only one crosses workspaces

There are currently two Invoke pipeline activities, and the difference is not cosmetic (Invoke pipeline activity[15]). Invoke pipeline (Legacy) supports only Fabric pipelines in the same workspace as the parent pipeline, can only monitor the parent pipeline, and cannot invoke Azure Data Factory or Synapse pipelines. The current Invoke pipeline activity invokes pipelines across Fabric workspaces, invokes them from Azure Data Factory or Synapse, and monitors child pipelines.

The current activity requires a Connection object held in the Fabric credentials store, and its Authentication kind offers three options: organizational account, service principal, or workspace identity. You then choose the target workspace and the pipeline within it. Using a workspace identity adds two prerequisites: the tenant setting Service principals can call Fabric public APIs has to be enabled because it is off by default, and when the target pipeline lives in another workspace, the identity needs at least Contributor access to that workspace.

Wait on completion decides the parent's flow. Leave it on when a following parent activity consumes the child's output, because the parent would otherwise move on before the child produced anything. Turn it off when the child can run independently, in which case the invoked pipeline executes in parallel with the activities that follow it in the parent pipeline's execution flow.

The figure below shows the nesting this produces. A Lookup activity and a ForEach activity, both from the control flow section above, sit inside the parent pipeline; inside the loop, a Notebook activity reaches the notebook with its parameter cell and session tag from the previous section, and an Invoke pipeline activity reaches a child pipeline in any workspace through its connection. That nesting is the reusable shape: the parent decides what to iterate over, and each child stays a single-purpose item.

Parent pipeline Lookup activity ForEach activity Notebook activity Invoke pipeline activity Notebook parameter cell, session tag Child pipeline any workspace, via connection
Parent-and-child orchestration: a ForEach inside a parent pipeline drives a Notebook activity and an Invoke pipeline activity.

The Dataflow activity

Running a Dataflow Gen2 as a pipeline step is what turns a standalone low-code transformation into part of an orchestrated workflow, and the activity carries a few settings that exist nowhere else.

The Dataflow activity runs a Dataflow Gen2 from a pipeline: select an existing workspace and dataflow on the Settings tab (Dataflow activity[5]). Where the selected item is a Dataflow Gen2 with CI/CD support and public parameters mode enabled, a Dataflow parameters section appears listing the available parameters, their types, and their default values. Required parameters are marked with an asterisk, cannot be deleted from the grid, and must be passed for the dataflow to run; optional parameters can be selected and removed.

Two operational notes save time later. Running a Dataflow Gen2 (CI/CD) successfully requires Member or higher access to the workspace plus access to all connections the dataflow uses, and missing either produces run failures. And the error information shown in the Dataflow activity can be incomplete or partial, so troubleshooting starts in the Dataflow Gen2 (CI/CD) recent runs view where full logs are downloadable, not in the pipeline output pane. Diagnosing failures across item types is covered on Identify and resolve errors.

The pattern to remember: the activity starts the dataflow and reports a summary, while the dataflow item itself remains the place where the run is actually observable.

The Semantic model refresh activity

The last step of the nightly job from the opening section is the one that makes the loaded data visible to report readers, and it is an activity like any other.

The Semantic model refresh activity connects to a Power BI semantic model and refreshes it from a pipeline, which is what closes the loop on a nightly load (Semantic model refresh activity[16]). It works only with semantic models you create yourself, and the workspace needs Power BI Premium, Premium Per User, or Power BI Embedded capacity.

By default the activity runs a full refresh when triggered from the pipeline. You can narrow it by picking only the tables and the partitions to refresh instead of the whole model, and both selections accept pipeline expressions, so the scope itself can be parameterized per run.

The Advanced section holds the execution controls. Wait on completion is on by default, which makes the activity wait until the refresh finishes before the pipeline moves on. Max parallelism, Retry Count, and Semantic model refresh timeout tune how the refresh executes, and a commit mode choice decides whether the refresh commits all at once (Transactional) or in batches (Partial Batch).

With these two activities the orchestration surface is complete: a pipeline chooses and sequences the work, a schedule or an event starts it, parameters and expressions carry the values, dependency conditions and control flow shape the graph, and the Invoke pipeline, Notebook, Dataflow, and Semantic model refresh activities reach out to the other items.

Exam-pattern recognition

DP-700 tests orchestration mostly through short scenarios that hand you a requirement and four plausible-looking configurations. The stems repeat, and so do the distractors, so recognizing the shape is worth more than recalling any single setting.

"Processing must start when a file is uploaded." Any answer that sets a schedule is wrong regardless of how short the interval is. The requirement names an event, so the answer is an event-based trigger built on eventstreams and an Activator rule. The distractor is seductive because a one-minute schedule would eventually notice the file; it is wrong because it polls rather than reacts.

"A low-code team maintains this transformation" versus "the source is a 500 GB file." These two stems point in opposite directions from the same three-item menu. Maintainability by non-programmers selects Dataflow Gen2; volume and custom algorithms select a notebook. Picking the notebook for the first stem is technically capable and organizationally wrong, which is exactly the trade being tested.

"Several existing pipelines must run in order, with a failure path." Only a data pipeline coordinates other items. Dataflow Gen2 offered as the coordinator is the standard wrong answer, and it is wrong on capability rather than on preference.

"The value must change as the loop runs." Parameter versus variable. A parameter's value stays the same throughout the run; anything incremented, appended to, or flipped is a variable.

"The activity property needs an integer." Interpolation versus direct reference. @{...} returns a string, and the direct @ form preserves the native type. The distractor wraps the value in @{...} and describes it as the safer option.

"The next activity consumes the child pipeline's output." Wait on completion stays enabled. Disabling it is the trap, and the answer text usually justifies it as improving parallelism.

"The pipeline to call is in another workspace." The current Invoke pipeline activity with a Fabric connection. Invoke pipeline (Legacy) is workspace-local and is the near-miss answer.

"Notebooks in this pipeline should share a Spark session." Session tags and the workspace high concurrency setting. An answer that sets only the tags is incomplete, and incomplete is wrong.

"Notify the team when the test run fails." Schedule failure notifications never cover on-demand runs, and the recipient list is item-wide rather than per-schedule. Watch which of those two facts the question actually turns on.

"Only some tables in the model need refreshing." The Semantic model refresh activity's table and partition selection, not a full model refresh and not a separate pipeline.

"Retry only on throttling." A retry condition matching Error code 429, not an unconditional retry count, and not a ForEach wrapper around the activity.

"The pipeline must run indefinitely." A far-future end date, because there is no open-ended schedule. An answer offering a blank end date describes a configuration that does not exist.

One reading habit helps more than memorizing any single mapping. Identify which of three questions the stem is really asking: which item does the work, what starts the run, or how the steps are wired together. Distractors are usually correct answers to one of the other two.

Choosing the Fabric item for the work

Decision factorDataflow Gen2Data pipelineNotebook
Authoring surfacePower Query visual connectors and transformationsActivity canvas with control flowCode in Python, Scala, SQL, or R
Primary jobClean, combine, and reshape dataSequence and coordinate other workDistributed processing on Apache Spark
Strongest fitLow-code transformation maintained by non-programmersRecurring movement and multi-step orchestrationVery large files, custom algorithms, logic beyond Power Query
Coordinates other itemsNoYes, through Notebook, Dataflow, and Invoke pipeline activitiesYes, through NotebookUtils notebook chaining and DAGs
Runs inside a pipelineYes, as a Dataflow activityYes, as an Invoke pipeline activityYes, as a Notebook activity
Started by an Activator ruleYesYesYes

Decision tree

Coordinate several items as one workflow? Needs custom code or Spark-scale processing? Target on the Dataflow Gen2 destination list? Data pipeline sequences the other items Notebook Python, Scala, SQL, R on Spark Dataflow Gen2 Power Query, fixed destinations Yes No Yes No Yes No Either way, a pipeline can invoke the chosen item as an activity

Sharp facts the exam loves — give these one last read before exam day.

Cheat sheet

Sharp facts the exam loves — scan these before test day.

Dataflow Gen2 fits visual low-code ingestion and transformation

Choose Dataflow Gen2 when Power Query's visual connectors and transformations can clean, combine, and reshape data without custom code. It can load multiple supported destinations and can also run as an activity inside a larger pipeline.

Trap Choose a notebook solely because a standard Power Query transformation must be maintained by low-code users.

7 questions test this
A data pipeline fits recurring movement and multi-step orchestration

Choose a Fabric data pipeline when the solution must coordinate copies, stored procedures, scripts, notebooks, dataflows, or conditional control flow as one workflow. Pipelines also provide connector and gateway-based movement for sources that are not directly suitable for Spark code.

Trap Use a Dataflow Gen2 item alone to coordinate several existing child pipelines with success and failure dependencies.

8 questions test this
A notebook fits code-first distributed Spark processing

Choose a Fabric notebook when ingestion or transformation needs custom Python, Scala, SQL, or R logic and Apache Spark's distributed processing. This is a strong fit for very large files, custom algorithms, or transformations that exceed the practical expressiveness of Power Query.

Trap Choose Dataflow Gen2 for a 500-GB file solely to obtain Spark's highest-throughput parallel file processing.

7 questions test this
A Notebook activity embeds Spark processing inside pipeline control flow

Use a Notebook activity when custom Spark code is required as one step of an orchestrated workflow. The pipeline supplies scheduling, dependencies, retries, and parameter passing, while the notebook remains responsible for its code-first data processing.

Trap Rewrite every pipeline activity into one notebook because pipelines cannot invoke Fabric notebooks.

8 questions test this
A fixed pipeline schedule requires start, end, frequency, and time zone

Fabric pipeline schedules are bounded rather than open ended, so configure both start and end dates along with frequency and time zone. A pipeline can have as many as 20 schedules, each with independent timing.

Trap Leave the end date blank to create a permanently open-ended Fabric pipeline schedule.

6 questions test this
Interval-based schedules expose nonoverlapping window boundaries to expressions

An interval-based schedule defines fixed nonoverlapping execution windows and supplies Window start time and Window end time as trigger parameters. Time-slice monitoring and backfill are not available for these interval schedules.

Trap Use an interval schedule because it automatically backfills every missed time window.

7 questions test this
Event triggers start pipelines in response to matching platform events

Use an event-based run when a pipeline must react to file, job, or workspace events rather than poll on a clock. Storage event setup uses Fabric eventstream and Activator capabilities and creates a Reflex item that holds the active trigger rule.

Trap Create a one-minute fixed schedule to satisfy a requirement that ingestion start specifically when a file arrives.

7 questions test this
Storage trigger file and folder metadata can drive dynamic pipeline expressions

Data Factory derives built-in file-name and folder-path trigger parameters from a storage event and exposes them in the expression builder. Use null-safe pipeline references when manually testing because an on-demand run has no storage event metadata.

Trap Expect a manual pipeline run to populate the same event file-name parameter as a BlobCreated trigger.

6 questions test this
Pipeline schedule failure notifications apply only to scheduled runs

Add users or groups to a pipeline schedule's Failure notifications when email is required after that scheduled run fails. Those schedule notifications are not sent for on-demand executions.

Trap Configure schedule failure notifications as the email mechanism for every manual test run.

4 questions test this
Managing schedules and failure notifications requires write-level access

To create or modify an item schedule or configure its failure notifications, a principal needs at least workspace Contributor or Write permission on the item. Viewer-only access permits inspecting existing schedules but not changing them.

6 questions test this
Failure notification recipients apply across every schedule for an item

Configure scheduled-run failure notification recipients at the Fabric item level. That recipient configuration applies to all schedules on the item rather than to only one selected schedule.

Activator rules start supported Fabric jobs and pass matching typed parameters

An Activator rule can start a pipeline, Dataflow, notebook, Spark job definition, user data function, or copy job when its condition is met. For supported parameterized items, define the Activator parameter name and type exactly as in the target and pass manual or dynamic values; copy jobs do not accept parameters.

Pipeline parameters provide run inputs that remain fixed during execution

Define typed parameters to reuse one pipeline with different external values on separate runs. A parameter value is immutable for the duration of a run, while variables are the appropriate construct for state that must change as activities execute.

Trap Use a pipeline parameter as a counter that a ForEach activity increments during the run.

5 questions test this
String interpolation converts a pipeline expression result to text

An expression referenced directly can preserve its native type, but an expression wrapped in @{...} interpolation always returns a string. Use direct expression form when an activity property requires a number, Boolean, array, or object rather than text.

Trap Wrap an integer parameter in string interpolation to guarantee that the activity receives an integer.

5 questions test this
Notebook activity parameter discovery reads the notebook parameter cell

A Notebook activity can auto-populate names, types, and defaults only for variables defined in the selected notebook's parameter cell. Pipeline authors can override those base values, and selecting a different notebook refreshes the discovered list.

Trap Expect the activity to auto-populate every variable declared in arbitrary notebook cells.

3 questions test this
Notebook session tags reuse Spark only when pipeline high concurrency is enabled

Assign the same session tag to compatible Notebook activities to reuse an existing Spark session and reduce startup time. The workspace setting for high concurrency on pipelines running multiple notebooks must be enabled; otherwise the tag does not provide session reuse.

Trap Set identical session tags while leaving high concurrency disabled and expect all notebook activities to share Spark.

4 questions test this
Invoke pipeline (Legacy) can wait for a child or let parent execution continue

Configure the Invoke pipeline (Legacy) activity to wait on completion when downstream parent activities depend on the child pipeline's completed output. Clear wait on completion when the child can execute independently: the invoked pipeline then runs in parallel with the activities that follow in the parent flow. Current documentation binds this wait/continue choice to the LEGACY activity's settings only; the modern Invoke pipeline activity's documented settings (Type, Connection, Workspace, Pipeline) carry no wait toggle.

Trap Clearing wait on completion on the Legacy activity when the next parent activity must consume the child pipeline's completed output.

1 question tests this
ForEach controls parallelism across items while preserving each item's child order

ForEach evaluates an item collection and exposes the current element through @item(). Select Sequential to process items one at a time; otherwise Batch count limits concurrent items, while the configured child activities still run sequentially for each individual item.

Trap Clear Sequential to make all child activities for one item execute simultaneously.

3 questions test this
Activity retries handle transient failures before control moves to a failure path

Configure retry count and interval on the failing activity when intermittent connectivity or throttling should be retried without custom orchestration. For supported activity types, optional retry conditions can restrict retries by error code, message, or failure type so permanent authentication errors do not consume unnecessary attempts.

Trap Add a ForEach loop around a Copy activity solely to retry a transient connection failure.

3 questions test this
NotebookUtils orchestrates reusable notebook chains and DAGs

Use the NotebookUtils notebook module to run and chain notebooks and receive exit values from them. It also supports parallel DAG execution and cross-workspace notebook references when notebook-driven orchestration is required.

3 questions test this
Pipeline dependency conditions select the required upstream outcome

Use On success or On failure when a downstream activity must run only for that respective upstream result, On completion when it must run after the upstream activity regardless of outcome, and On skip when it must run only if the upstream activity is skipped.

5 questions test this
Choose pipeline control flow by branch shape and loop condition

Use If Condition for true and false branches, Switch for multiple branches selected by one expression, and Until to repeat nested activities until its condition becomes true.

4 questions test this
Lookup and Get Metadata outputs drive data-dependent pipeline control flow

Use Lookup to return configuration rows or values and Get Metadata to return properties such as existence, structure, or column count. Reference their outputs as dynamic content in succeeding activities to drive branches and loops instead of hard-coding orchestration decisions.

Secure input and output prevent activity values from appearing in logs

Enable Secure input to suppress an activity's input from pipeline logs and Secure output to suppress its output. Apply the corresponding setting wherever sensitive parameters, credentials, or returned values would otherwise be exposed in run logging.

4 questions test this
The current Invoke pipeline activity uses a connection for cross-workspace calls

Use the current Invoke pipeline activity, not Invoke pipeline (Legacy), to call a Fabric pipeline in another workspace. Its Fabric connection can authenticate with an organizational account, service principal, or workspace identity; the legacy activity is limited to the parent pipeline's workspace.

3 questions test this
Pipeline schedules bind exact parameter names to static or library values

A schedule can pass either a direct static value or a Variable Library reference to each pipeline parameter at run time. The schedule's parameter name must exactly match the pipeline definition; a mismatched name is ignored.

3 questions test this
Semantic model refresh activity controls both refresh scope and execution behavior

A Semantic model refresh activity can target selected tables or partitions when the pipeline should process only part of a model. Configure its retry count, maximum parallelism, and refresh timeout to control execution behavior for that scoped refresh.

References

  1. Activity overview
  2. What is Power Query?
  3. Differences between Dataflow Gen1 and Dataflow Gen2
  4. Fabric decision guide: copy activity, dataflow, Eventstream, or Spark
  5. Dataflow activity
  6. Notebook activity
  7. Run, schedule, or use events to trigger a pipeline
  8. Job Scheduler in Microsoft Fabric
  9. Trigger Fabric items
  10. Parameters
  11. Expression language
  12. ForEach Activity
  13. Activity Retries in Microsoft Fabric Data Factory
  14. NotebookUtils (former MSSparkUtils) for Fabric
  15. Invoke pipeline activity
  16. Semantic model refresh activity in Data Factory for Microsoft Fabric