Implementing Lakeflow Jobs

• Each task chooses job compute, serverless, or an existing all-purpose cluster
• Job parameters are key-value pairs passed to every task
• Set Run as, max concurrent runs, and tags when configuring a job
• Tasks can share one job cluster to cut startup cost
• A notebook task reads job parameters with dbutils.widgets.get
• Lakeflow Jobs supports scheduled, file-arrival, table-update, continuous, and manual triggers
• Continuous keeps a job always-on; file arrival launches a run per batch of files
• A table-update trigger runs a job when a monitored table changes
• File-arrival triggers monitor a Unity Catalog external location or volume
• A scheduled trigger uses a quartz cron expression with a timezone
• Pause a schedule to stop runs without losing the job definition
• Queueing controls what happens when a scheduled run overlaps a running one
• Notifications attach at the job or the individual task level
• Notifications go to email or admin-configured system destinations
• Duration thresholds warn when a run is late or slow
• Continuous jobs auto-restart the whole run with exponential backoff
• SDP pipelines auto-recover; non-continuous jobs use per-task retries
• Restart run cancels the active run and resets the backoff period
• When a Run if condition is evaluated an Excluded upstream counts as successful, while Upstream failed or Upstream canceled counts as failed
• Cancelling a task run propagates downstream and still runs the tasks whose Run if condition handles failure
• Downstream tasks depend on the For each task, never on its nested task
• A For each task cannot contain another For each task, and it fails outright when the upstream task supplying its inputs is disabled
• A stage another team owns belongs in its own job invoked by a Run Job task rather than copied as tasks into every consumer job
• Run Job chains may not be circular, directly or indirectly, and may not be nested arbitrarily deep
• Whether a caller can trigger a child job and override its parameters is decided by the permission the caller's run-as identity holds on the CHILD job
• A value an operator must vary per run belongs in a job parameter, because a static task parameter can only be changed by editing the task definition
• A dynamic value reference is the layer that lets a task consume run metadata or another task's value at run time instead of a hard-coded literal
• Only a task value can carry a value the job could not know before it started, and it flows forward along task dependencies only

Unlock with Premium — includes all practice exams and the complete study guide.