Ingesting & Processing Data

A data pipeline moves raw data from a source to a usable result, and at 25% this is the heaviest-weighted domain on the exam, so the model that runs through it is worth pinning down. Two questions decide almost every answer here. First, the latency contract: how fresh must the output be? Sub-second to seconds freshness means a streaming pipeline (Pub/Sub ingesting, Dataflow processing each event as it arrives); a next-morning or weekly contract means a batch pipeline that processes a bounded set on a schedule for less cost. Second, where the data already lives and who is building it: data already in BigQuery with a SQL team stays in BigQuery; existing Spark or Hadoop code lifts onto Dataproc; new custom logic goes to Dataflow on Apache Beam; analysts who do not code reach for Cloud Data Fusion. The classic trap is reaching for the tool you like before you have stated the freshness requirement and located the data, so the requirement, not the preference, names the service.

Read the three subtopics as the life of one pipeline, in order. Planning Pipelines is the design step: you fix the source, the sink, and the latency contract on paper, then let those choices select batch versus streaming and the processing service (Dataflow, Dataproc, Cloud Data Fusion, or BigQuery SQL), and you plan networking and encryption into the design rather than after the first failure. Building Pipelines is the hands-on step that wires three jobs in data-flow order: acquire the data (Pub/Sub for events, Datastream or a transfer service for bulk), transform it (the core, where windowing and data cleansing live for streams), then optionally enrich it with a model prediction. Deploying & Operationalizing is the ship step: orchestrate the runs (Cloud Composer for dependency-rich DAGs, Workflows for light service chaining, Cloud Scheduler as the bare cron trigger) and promote the code from dev to prod through CI/CD with parameterized artifacts. Plan decides what to build, build writes the logic, deploy automates and ships it.

Two instincts settle most close calls in this domain. The first is that orchestration is a different decision from processing: a single transform job is not a pipeline, and a pipeline is several steps run in order with retries and a schedule. Pick the processor by the data and the latency contract, then pick the orchestrator separately by how many dependent steps it must coordinate; never make the processor double as the orchestrator. The second is to run the work where the data already is. If the data sits in BigQuery and the team writes SQL, transform it in place with Dataform or BigQuery SQL and run inference with BigQuery ML so nothing leaves the warehouse, rather than exporting it to a separate engine. The exam rewards the answer with the lowest operational burden that still meets the freshness contract, which usually means the most-managed, serverless option that keeps the data home.