Building Pipelines — PDE Study Guide

The build: acquire, transform, enrich

A retail team needs yesterday's clickstream joined to the product catalog and scored for churn risk, landing in BigQuery by 7am. That one request is the whole subtopic: get the data in, reshape it, add a prediction. Building a pipeline is those three jobs wired in the order data moves, and the Professional Data Engineer exam tests which Google Cloud service owns each job for a given scenario.

Acquire brings raw data into Google Cloud. Real-time events arrive through Pub/Sub^[1]; bulk and recurring loads use a managed mover such as Datastream^[2] for database change capture, BigQuery Data Transfer Service^[3] for scheduled loads into BigQuery, or Storage Transfer Service^[4] for object data into Cloud Storage. Transform is the core job: clean, reshape, join, and aggregate. Dataflow^[5] runs custom code for stream or batch, Dataproc^[6] runs existing Apache Spark and Hadoop jobs, Cloud Data Fusion^[7] builds transforms visually, and Dataform^[8] transforms data already in BigQuery with SQL. Enrich adds a model prediction as a column, in BigQuery with BigQuery ML^[9] or mid-stream by calling a Vertex AI endpoint.

Keep two neighbours straight, because the exam dangles them as distractors. Planning a pipeline is the design-time choice of which engine and why, covered in the planning subtopic; deploying and operationalizing is scheduling the run, retrying, and promoting code through environments with Cloud Composer, the next subtopic. This page is the build itself: the transformation logic and which service implements it. The rest of the page walks each job in that data-flow order, so a term defined in acquisition is reused, not re-explained, in transform.

The three jobs of a pipeline build in data-flow order, with the Google Cloud service that owns each.

Acquiring and importing the data

Acquisition is choosing the right intake for the source's shape: a never-ending stream of events, a continuously changing database, or a finite batch of files or rows. Match the mover to the source and the answer follows.

Pub/Sub for real-time events

Pub/Sub^[1] is the fully managed, serverless messaging service that decouples publishers from subscribers and is the real-time ingestion layer of a streaming pipeline: it buffers and absorbs spikes in event volume so a downstream processor is never overrun. The canonical streaming build is Pub/Sub ingests, Dataflow processes, BigQuery stores. Reach for Pub/Sub whenever the source emits a continuous flow of events (IoT telemetry, app logs, clickstream) that must be processed as it arrives.

Datastream for database change capture

Datastream^[2] is a serverless change data capture (CDC) and replication service that keeps a destination in sync with a source database at minimal latency by streaming inserts, updates, and deletes rather than re-loading whole tables. Sources include Oracle, MySQL, SQL Server, and PostgreSQL; destinations are BigQuery and Cloud Storage. Its signature use is keeping a BigQuery analytics copy current with an operational database. Do not confuse it with Database Migration Service, which lands a managed Cloud SQL or AlloyDB database you then operate; Datastream feeds analytics, DMS migrates the database itself.

Managed batch loaders

For bulk and recurring loads, pick by destination:

Mover	Destination	Source examples	Signature use
BigQuery Data Transfer Service	Always BigQuery	Redshift, Teradata, Snowflake, Cloud Storage, Amazon S3, Google Ads, GA4	No-code scheduled loads into BigQuery
Storage Transfer Service	Cloud Storage	Amazon S3, Azure Blob, HDFS, on-prem file systems, other GCS buckets	Object data into Cloud Storage, optimized for over 1 TiB

BigQuery Data Transfer Service^[3] automates scheduled, managed, no-code movement into BigQuery; a transfer can be a one-time backfill or a recurring scheduled load. Storage Transfer Service^[4] moves object data into Cloud Storage with automatic retries and is optimized for transfers larger than 1 TiB; for a small one-off copy use gcloud storage instead of standing up a transfer job. The rule: events stream through Pub/Sub, a changing database flows through Datastream, scheduled bulk lands through the transfer service that matches the destination.

Choosing the acquisition service by the source's shape: real-time stream, changing database, or scheduled bulk load.

Transforming and cleansing: batch vs streaming

The transform step turns raw input into the records consumers need, and the same job also cleanses: parsing, deduplicating, standardizing formats, and rejecting or routing bad rows. The headline that unifies batch and streaming is that they share one programming model, so the real choice is the engine, and the engine follows the team's code and where the data already lives.

One model, two input kinds

Dataflow^[5] runs Apache Beam^[5] pipelines and uses the same programming model for batch and stream analytics. A Beam pipeline reads from a source, applies a chain of transforms, and writes to a sink; the pipeline definition does not specify how it runs, a runner (here the Dataflow runner) executes it on the platform. The only difference between batch and streaming is the input: batch processes a bounded dataset that is already complete when the job starts, while streaming processes an unbounded dataset that never ends. Because the code is the same, you can start with a batch pipeline and run the identical logic on a stream by swapping the source, which is why the exam frames Dataflow as the unified, serverless processing engine.

Dataproc for existing Spark and Hadoop code

Dataproc^[6] is managed open-source Apache Spark, Hadoop, Hive, and Presto/Trino with the same tools and APIs as on-premises, so existing Spark or Hadoop jobs lift and shift with minimal redevelopment. Clusters create, scale, and shut down in about 90 seconds and read from Cloud Storage, BigQuery, and Bigtable, which decouples storage from compute so a cluster can be ephemeral and billed only for the run. The decision rule against Dataflow is about code, not capability: choose Dataproc when you are porting existing OSS code and want to keep your Spark or Hadoop jobs, choose Dataflow when you are writing new serverless code in the Beam model.

Visual and SQL transforms

Not every builder writes pipeline code. Cloud Data Fusion^[7] builds ETL pipelines through a drag-and-drop visual interface with no code, and includes Wrangler, an interactive tool that parses, filters, and reshapes data with point-and-click directives; it executes on ephemeral Dataproc clusters it provisions per run. Dataform^[8] manages the ELT pattern inside BigQuery: the data is already loaded and Dataform runs the transform step as version-controlled SQL. Use Data Fusion when analysts build visually, Dataform or plain BigQuery SQL when the data is already in BigQuery and the team is SQL-fluent.

Cleansing and the dead-letter pattern

Cleansing is not a separate product, it is logic inside whichever transform you chose. The durable pattern for code pipelines is the dead-letter route: a Dataflow pipeline sends records that fail parsing or validation to a side destination (a separate table or topic) instead of throwing, so one malformed row does not crash the whole job and the bad records are kept for inspection. Express value-cleansing as Beam transforms in Dataflow, Wrangler directives in Data Fusion, or SQL CASE/SAFE_CAST in Dataform; the principle is the same, fail the row, not the run.

Apache Beam runs one set of transforms over bounded (batch) or unbounded (stream) input; good rows go to the sink, failed rows to a dead-letter.

Streaming transforms: windows, watermarks, triggers, late data

Aggregating an unbounded stream raises a question batch never asks: when is a group of data complete enough to compute on? You cannot wait for the end because there is no end. Streaming answers with four cooperating pieces, and the exam tests each by name.

Windows slice the stream

A window divides the unbounded stream into finite intervals you can aggregate over. There are three kinds. A tumbling window (Google also calls it a fixed time window) is a consistent, disjoint interval: thirty-second windows partition the stream into back-to-back, non-overlapping buckets, so each element lands in exactly one window. A hopping window (a sliding time window) is a consistent interval that can overlap, defined by a window length plus a shorter period at which a new window starts, so an element can belong to several windows at once; use it for a moving average like "the last 5 minutes, recomputed every 1 minute." A session window contains elements within a gap duration of one another: it groups bursts of activity per key and closes when a gap of inactivity passes, which fits per-user clickstream sessions where the boundaries depend on the data, not the clock.

Watermarks decide when a window is done

Data is not guaranteed to arrive in time order or at predictable intervals, so the runner needs an estimate of completeness. A watermark is a threshold that indicates when Dataflow expects all of the data in a window to have arrived. When the watermark passes the end of a window, the runner treats that window as ready and, by default, emits its result.

Triggers decide when to emit

A trigger determines when to emit aggregated results as data arrives. The default trigger fires once, when the watermark passes the end of the window. The Apache Beam SDK also offers triggers based on processing time (emit early, on a wall-clock interval, so you get a partial result before the window closes) and on element count (data-driven, emit after N elements). Early triggers trade completeness for latency: you see a result sooner but it may be revised.

Late data arrives after the watermark

An element whose event time falls inside a window but that arrives after the watermark has passed that window's end is late data. By default late data is dropped, but you can configure allowed lateness so the window stays open for a grace period and recomputes when a straggler arrives; combined with an accumulating trigger, the window re-emits an updated result. The exam pattern is a stem describing out-of-order events and asking how to keep stragglers, the answer pairs a watermark with allowed lateness, not a bigger window.

Tumbling windows are disjoint, hopping windows overlap, session windows close after a gap; the watermark marks completeness and the trigger fires the emit.

AI enrichment and reading the exam stem

The last build job adds a model's prediction as a column: a churn flag, a forecast, a sentiment score, computed as part of the pipeline so consumers read enriched data directly. Where the inference runs follows the same axis as the transform engine, where the data lives and who is building.

In-warehouse inference with BigQuery ML

BigQuery ML^[9] lets you create and run machine-learning models inside BigQuery using standard SQL, so no data leaves the warehouse. You train with CREATE MODEL and score with the ML.PREDICT function over a query, which returns the input rows plus a predicted column. It fits when the data already lives in BigQuery and the team is SQL-fluent rather than ML engineers, and it is the enrichment answer for a batch, SQL-driven transform. BigQuery ML can also call a registered remote model, including a Vertex AI model or a Gemini large language model, through the same ML.PREDICT or ML.GENERATE_TEXT interface, so an LLM enrichment such as summarizing or classifying text stays inside the SQL pipeline.

Mid-stream inference from Dataflow

When enrichment must happen as events flow, call a deployed Vertex AI endpoint from inside a Dataflow transform: each element is sent for a prediction and the scored element continues down the pipeline. This is the streaming counterpart to ML.PREDICT, and it is the right answer when the scenario scores data in real time (fraud checks on a transaction stream, for example) rather than in a scheduled batch over a BigQuery table.

Reading the stem

These scenarios resolve to a small set of tells. A real-time event stream with windowed aggregation is Pub/Sub plus Dataflow; an answer that offers Cloud Composer as the processor is the orchestration distractor (Composer schedules jobs, it does not transform data). "We already have Spark/Hadoop jobs to migrate" points to Dataproc, not a Dataflow rewrite. "Analysts build the pipeline without coding" is Cloud Data Fusion. "Transform data already in BigQuery with SQL" is Dataform or BigQuery SQL, not exporting to an external engine. "Keep BigQuery current with our production database" is Datastream CDC, not a full reload. "Add a prediction with SQL where the data sits" is BigQuery ML; "score each event as it streams" is a Vertex AI call from Dataflow. Match the tell to the service and the distractors fall away.

AI enrichment splits on timing: in-warehouse SQL inference with BigQuery ML, or a mid-stream Vertex AI call from Dataflow.

Choosing the transform engine by team skill and where data lives

Engine	Programming model	Stream + batch	Best fit
Dataflow	Apache Beam (Java/Python/Go), serverless	Both, one model	Custom or real-time logic, streaming windowing
Dataproc	Apache Spark / Hadoop / Hive / Presto, clusters	Batch (Spark Structured Streaming possible)	Porting existing OSS jobs with minimal rewrite
Cloud Data Fusion	Visual drag-and-drop, no code	Both (via pipelines)	Analysts building ETL without writing code
Dataform / BigQuery SQL	SQL / SQLX, in-warehouse ELT	Batch (scheduled)	Transforming data already loaded in BigQuery

Decision tree

Cheat sheet

Sharp facts the exam loves — give these one last read before exam day.

Cheat sheet

Sharp facts the exam loves — scan these before test day.

Pub/Sub ingests, Dataflow processes, BigQuery stores: the streaming build

The canonical Google Cloud streaming pipeline pairs Pub/Sub as the real-time ingestion layer with Dataflow as the processing stage, writing the result to BigQuery. Pub/Sub is fully managed, serverless messaging that decouples publishers from subscribers and buffers spikes in event volume so the processor is never overrun; Dataflow then applies the transform and aggregation logic. When a stem describes events arriving continuously and needing windowed aggregation, this is the build.

Trap Naming Cloud Composer as the stream processor; Composer orchestrates multi-step workflows, it does not transform the data itself.

Apache Beam runs one model for batch and streaming; only the input differs

Dataflow runs Apache Beam pipelines and uses the same programming model for batch and stream analytics, so you write the transform logic once. The only difference is the input: batch processes a bounded dataset that is complete when the job starts, while streaming processes an unbounded dataset that never ends. Because the code is identical, you can run the same pipeline on a stream by swapping the source for Pub/Sub.

Dataflow does not specify how it runs; a runner executes the pipeline

An Apache Beam pipeline defines what to compute, not how to execute it; a runner carries it out on a specific platform, and the Dataflow runner is the managed, serverless, autoscaling runner on Google Cloud. This separation is why the same Beam code can target different runners. Dataflow provisions and manages all the infrastructure, so there are no clusters to size.

Choose Dataproc to port existing Spark/Hadoop, Dataflow to write new serverless code

Dataproc is managed open-source Apache Spark, Hadoop, Hive, and Presto/Trino with the same tools and APIs as on-premises, so existing OSS jobs lift and shift with minimal redevelopment. The decision against Dataflow is about code, not capability: pick Dataproc when migrating Spark or Hadoop jobs you already have, and Dataflow when authoring new pipelines in the Beam model. Dataproc clusters create and shut down in about 90 seconds and read from Cloud Storage, BigQuery, and Bigtable, so they can be ephemeral and billed per run.

Trap Rewriting working Spark/Hadoop jobs into Apache Beam for Dataflow when Dataproc would run them as-is with minimal change.

Cloud Data Fusion is the code-free, visual ETL builder

Cloud Data Fusion builds ETL pipelines through a drag-and-drop visual interface with no code, which fits when the builders are data analysts rather than software engineers. It includes Wrangler, an interactive tool that parses, filters, and reshapes data with point-and-click directives for cleansing. Under the hood it executes pipelines on ephemeral Dataproc clusters it provisions per run, so you pay only for the run.

6 questions test this

Dataform runs the transform inside BigQuery with version-controlled SQL

Dataform manages the ELT pattern inside BigQuery: the data is already loaded and Dataform runs the transform step as version-controlled SQLX (SQL plus metadata), compiling the dependencies into a DAG (directed acyclic graph, the ordered task-dependency map) it runs in BigQuery. Choose it when the data is already in BigQuery and the team is SQL-fluent, so you transform in place instead of moving data to a separate engine. For one-off SQL transforms, plain BigQuery SQL serves the same role without the project structure.

Tumbling (fixed) windows are disjoint; every element lands in exactly one

A tumbling window, which Google also calls a fixed time window, is a consistent, disjoint time interval: back-to-back, non-overlapping buckets such as 0:00-0:30 then 0:30-1:00. Each element belongs to exactly one window, which makes tumbling the right choice for periodic, non-overlapping aggregations like per-minute counts. Use it when each event should be counted once and only once.

Trap Using a tumbling window for a moving average; disjoint windows never overlap, so a rolling metric needs a hopping (sliding) window instead.

Hopping (sliding) windows overlap; an element can sit in several

A hopping window, also called a sliding time window, is a consistent interval that can overlap, defined by a window length plus a shorter period at which a new window starts. Because windows overlap, one element can belong to multiple windows, which is exactly what a moving average needs: a 5-minute window recomputed every 1 minute reports the last 5 minutes' value once per minute. The period controls how often you emit, the length controls how much history each emit covers.

Session windows close after a gap of inactivity, per key

A session window contains elements within a gap duration of one another and is keyed, so it groups a burst of activity and closes when a gap of inactivity passes without new data. The window boundaries come from the data, not the clock, which fits per-user clickstream sessions or activity bursts of unpredictable length. The gap duration, not a fixed size, is the tuning knob.

Trap Forcing a fixed-size window onto irregular per-user activity; session windows exist precisely because the boundaries depend on data gaps, not a clock interval.

A watermark marks when a window's data is expected to be complete

Because streamed data is not guaranteed to arrive in time order or at predictable intervals, the runner needs an estimate of completeness. A watermark is a threshold indicating when Dataflow expects all of a window's data to have arrived; when the watermark passes the end of a window, the runner treats it as ready and, by default, emits its result. The watermark is about completeness, not about when you choose to emit, which is the trigger's job.

Trap Confusing the watermark with the trigger; the watermark estimates completeness, the trigger decides when to emit results.

A trigger decides when to emit; default fires once at the watermark

A trigger determines when to emit aggregated results as data arrives. The default trigger fires once, when the watermark passes the end of the window, giving one complete result. The Apache Beam SDK also offers processing-time triggers (emit early on a wall-clock interval) and data-driven triggers (emit after a count of elements), which trade completeness for lower latency by producing partial, possibly revised results before the window closes.

Keep late data with allowed lateness, not a bigger window

Late data is an element whose event time falls inside a window but that arrives after the watermark has passed that window's end; by default it is dropped. To keep stragglers, configure allowed lateness so the window stays open for a grace period and recomputes when a late element arrives, and pair it with an accumulating trigger so the window re-emits an updated result. Widening the window changes the aggregation boundaries instead and does not address out-of-order arrival.

Trap Enlarging the window to capture out-of-order events; that changes the aggregation grouping, whereas allowed lateness is what keeps stragglers in their correct window.

Route failed records to a dead-letter so one bad row does not stop the run

Cleansing lives inside the transform, and the durable pattern for code pipelines is the dead-letter route: a Dataflow pipeline sends records that fail parsing or validation to a side destination (a separate table or topic) instead of throwing, so one malformed row does not crash the whole job and the bad records are preserved for inspection. The principle is fail the row, not the run, and it applies whether you cleanse with Beam transforms, Wrangler, or SQL.

Trap Letting a parse failure throw and abort the batch; a dead-letter sink isolates the bad record while the rest of the data keeps flowing.

6 questions test this

Datastream is change data capture; it keeps BigQuery in sync with a live database

Datastream is a serverless change data capture (CDC) and replication service that streams inserts, updates, and deletes from an operational database to a destination with minimal latency, rather than re-loading whole tables. Sources include Oracle, MySQL, SQL Server, and PostgreSQL; destinations are BigQuery and Cloud Storage. Its signature use is keeping a BigQuery analytics copy current with a production database, distinct from Database Migration Service, which lands a managed Cloud SQL or AlloyDB database you then operate.

Trap Choosing Database Migration Service to feed analytics; DMS migrates the database to operate it, while Datastream's CDC feeds an in-sync analytics copy.

BigQuery Data Transfer Service is no-code scheduled loads into BigQuery

BigQuery Data Transfer Service (DTS) automates scheduled, managed, no-code movement of data into BigQuery, where the destination is always BigQuery. Sources span warehouses and databases (Redshift, Teradata, Snowflake, Oracle), object stores (Cloud Storage, Amazon S3, Azure Blob), and SaaS feeds (Google Ads, GA4, YouTube, Salesforce). A transfer can be a one-time backfill or a recurring scheduled load, so reach for DTS when the requirement is repeatable ingestion into BigQuery without writing pipeline code.

10 questions test this

Storage Transfer Service moves objects into Cloud Storage, tuned for large transfers

Storage Transfer Service moves object data online into Cloud Storage from Amazon S3, Azure Blob Storage, HDFS, on-premises file systems via agents, and other Cloud Storage buckets, with automatic retries and scheduled or one-time runs. It is optimized for transfers larger than 1 TiB; for a small one-off copy use gcloud storage instead of standing up a transfer job. Pick the mover by destination: Storage Transfer lands objects in Cloud Storage, BigQuery DTS lands rows in BigQuery.

Trap Standing up Storage Transfer Service for a sub-1-TiB ad-hoc copy; gcloud storage is the lighter tool below its optimized threshold.

Enrich in-warehouse with BigQuery ML ML.PREDICT, no data leaving BigQuery

BigQuery ML lets you create and run machine-learning models inside BigQuery using standard SQL, so enrichment adds a predicted column without data leaving the warehouse. You train with CREATE MODEL and score with the ML.PREDICT function over a query, which returns the input rows plus the prediction. It fits a batch, SQL-driven transform when the data already lives in BigQuery and the team is SQL-fluent rather than ML engineers.

Score events mid-stream by calling a Vertex AI endpoint from Dataflow

When enrichment must happen as events flow rather than in a scheduled batch, call a deployed Vertex AI endpoint from inside a Dataflow transform: each element is sent for a prediction and the scored element continues down the pipeline. This is the streaming counterpart to BigQuery ML's ML.PREDICT, and it is the answer when the scenario scores data in real time, such as fraud checks on a transaction stream, instead of over a static BigQuery table.

Trap Reaching for BigQuery ML to score a live event stream; ML.PREDICT runs over data already in BigQuery, so real-time scoring calls a Vertex AI endpoint from Dataflow.

BigQuery ML can call a remote Gemini or Vertex model for LLM enrichment in SQL

Beyond models trained in BigQuery, BigQuery ML can invoke a registered remote model, including a Vertex AI model or a Gemini large language model, through ML.PREDICT or ML.GENERATE_TEXT. This keeps LLM enrichment such as summarizing or classifying text inside the SQL pipeline, so you add a generated column without exporting the data. It is the in-warehouse path to generative enrichment for SQL-fluent teams.

An Eventarc Cloud Storage trigger must sit in the bucket's region

The Eventarc trigger's location must match the location (region or multi-region) of the Cloud Storage bucket it watches. The destination Cloud Function/Cloud Run service can live in a different region, but a location mismatch on the trigger itself causes deployment errors; deploying the function in the same region too minimizes latency.

Trap Creating the trigger in the function's region instead of the bucket's region, which fails with a location-mismatch error.

7 questions test this

GCS Eventarc triggers need Pub/Sub Publisher on the storage agent plus Run Invoker

Cloud Storage triggers deliver events through Pub/Sub, so the Cloud Storage service agent must hold roles/pubsub.publisher on the project, and the trigger's service account needs roles/run.invoker (and roles/eventarc.eventReceiver) to invoke the target. Having only Cloud Functions Developer or Eventarc Event Receiver is not enough and yields permission-denied at deploy time.

Trap Granting only the Eventarc Event Receiver role and omitting Pub/Sub Publisher on the Cloud Storage service agent.

6 questions test this

Pick the right GCS event type: finalized for create/overwrite, archived for versioning, deleted for removal

google.cloud.storage.object.v1.finalized fires when an object is created or overwritten (a new generation). archived fires when a live version becomes noncurrent due to overwrite or deletion in a versioning-enabled bucket. deleted fires when an object is permanently removed. Match the event type to the exact change you need to react to.

Trap Using the deleted event to catch overwrites in a versioned bucket, when archived is the event for a version becoming noncurrent.

6 questions test this

Make event-driven functions idempotent because Eventarc delivers at-least-once

Cloud Functions triggered by events get at-least-once delivery, so the same event can arrive more than once, especially with retries enabled. Prevent duplicate writes by using the CloudEvent ID as an idempotency key: record or transactionally check it (e.g. in Firestore) before processing, or check whether the output already exists. Enable retries for reliability, and add an age check to stop endless retry loops on poison events.

Trap Enabling retries for reliability without an idempotency key, which turns transient failures into duplicate records.

8 questions test this

Fan out past the 10-notification GCS limit with one Pub/Sub topic

A Cloud Storage bucket allows only 10 notification configurations per event type, so the 11th direct-trigger function fails. The fix is one notification configuration that publishes to a single Pub/Sub topic, then have every consumer function subscribe to that topic, which supports far more subscribers and lets functions filter (e.g. by file extension) in code.

Trap Adding yet another per-function Cloud Storage notification, which hits the same 10-per-event-type ceiling.

5 questions test this

Join a small reference table into a big PCollection with a side input

When one side of a Beam join is much smaller than the other and fits in worker memory (e.g. an MB-scale catalog enriching TB-scale transactions), load the small one as a side input and look it up inside a ParDo. The side input is cached in memory and reused across ParDos with no shuffle. Use CoGroupByKey only when both datasets are too large to fit in memory.

Trap Using CoGroupByKey to attach a tiny reference table, forcing an expensive shuffle a side input avoids.

5 questions test this

Speed up Data Fusion joins by pushing them down to BigQuery

When a Cloud Data Fusion pipeline is slow because heavy JOINs run as Apache Spark on Dataproc, enable Transformation Pushdown so supported transformations (notably complex joins) execute in BigQuery instead. This improves performance for join-heavy pipelines without rewriting the pipeline logic.

Trap Scaling up the Dataproc cluster to speed slow Data Fusion joins, instead of pushing the joins down to BigQuery.

6 questions test this

Add a missing Data Fusion connector by deploying it from the Hub

Cloud Data Fusion ships a default plugin set; connectors that are absent from the Studio palette (Teradata, Salesforce, SAP, etc.) are added by deploying them from the Hub into your namespace, after which they appear in the palette in that namespace. No custom plugin development is needed for these pre-built connectors.

Trap Assuming a connector missing from the palette is unsupported, instead of deploying it from the Data Fusion Hub.

6 questions test this

Map Data Fusion stages to plugin categories: source, analytics (Joiner/Group By), sink

In a Cloud Data Fusion pipeline, source plugins read from databases and files, analytics plugins perform joins and aggregations (the Joiner plugin combines sources on a key, Group By performs sum/count/avg per group), and sink plugins write the output (e.g. to BigQuery). Joining two sources is an Analytics-category operation, not a source or sink one.

Trap Looking for a join under source or sink plugins, when joining lives in the Analytics category (Joiner).

6 questions test this

The build: acquire, transform, enrich

Acquiring and importing the data

Pub/Sub for real-time events

Datastream for database change capture

Managed batch loaders

Transforming and cleansing: batch vs streaming

One model, two input kinds

Dataproc for existing Spark and Hadoop code

Visual and SQL transforms

Cleansing and the dead-letter pattern

Streaming transforms: windows, watermarks, triggers, late data

Windows slice the stream

Watermarks decide when a window is done

Triggers decide when to emit

Late data arrives after the watermark

AI enrichment and reading the exam stem

In-warehouse inference with BigQuery ML

Mid-stream inference from Dataflow

Reading the stem

Choosing the transform engine by team skill and where data lives

Decision tree

Cheat sheet

References