Designing and Implementing Data Pipelines

• Pipeline order of operations follows the bronze-silver-gold medallion flow
• Lakeflow SDP infers execution order from declared dataset dependencies
• Separating ingest, transform, and serve stages lets each scale and recover independently
• Choose SDP for declarative incremental ETL with built-in quality and orchestration
• SDP manages infrastructure; a notebook task requires you to manage it imperatively
• Notebooks suit procedural or non-ETL logic that cannot be expressed declaratively
• A Lakeflow Job orchestrates a DAG of tasks linked by depends_on
• Control-flow tasks add branching, iteration, and nested jobs
• Pass data between tasks with task values and parameter references
• Tasks come in several types beyond notebooks
• A task retry policy automatically re-runs a failed task with a delay
• Run-if conditions let downstream tasks run even when an upstream fails
• Notebooks handle errors with try/except and surface status via dbutils.notebook.exit
• %run inlines a notebook; dbutils.notebook.run executes it as a separate scope
• Express precedence with Job tasks rather than deeply nested notebook calls
• dbutils.notebook.run enables conditional branching and returns a string
• SDP datasets are defined with pyspark.pipelines Python decorators
• Streaming tables ingest incrementally; materialized views recompute results
• Auto Loader is the recommended incremental file source inside SDP
• SDP can also be authored in SQL with CREATE OR REFRESH statements
• SDP development mode keeps compute warm; production mode restarts and retries
• SDP pipelines update in triggered or continuous mode
• A job that orchestrates a pipeline determines the execution mode and overrides the pipeline's own Pipeline mode setting
• pipelines.trigger.interval controls update frequency only under continuous execution, never the cadence of a triggered pipeline
• Setting the table property pipelines.reset.allowed to false stops a full refresh from ever clearing that one table
• Tables excluded from a failed-table update are not refreshed even when they depend on a failed table
• A repair run re-runs the unsuccessful tasks with the job's CURRENT settings, not the settings captured when the run started
• Onboarding another source as one more append flow keeps an existing streaming table updating incrementally, while folding that source into the target's defining query rewrites its single flow and forces the target to be rebuilt
• A history load that must happen exactly once is a ONCE flow, while a backfill location that keeps receiving late data needs an ordinary append flow left in place
• The flow, not the streaming table, owns the streaming checkpoint, and a flow's identity is its name

Unlock with Premium — includes all practice exams and the complete study guide.