Planning Pipelines
The planning sketch: source, sink, latency, network, keys
An e-commerce team wants a fraud-scoring pipeline live in two weeks. The first hour is not spent writing code; it is spent answering five questions on a whiteboard, and every answer constrains the next. Where does the data enter (the source)? Where must the result land for the people who use it (the sink)? How fresh does that result have to be (the latency contract)? What network do the workers run in, and what reaches them privately? Whose keys encrypt the data along the way? Planning a pipeline is filling in that sketch before you choose a tool, because the tool falls out of the answers rather than driving them.
The five pieces are not independent, and the order matters. The latency contract is the hinge: seconds-fresh forces a streaming source and a streaming processor, while next-morning-fresh allows a cheaper batch job. The source and sink then pin the ingestion and storage edges: a streaming source such as Pub/Sub feeds a streaming processor that writes to BigQuery or Bigtable; a file landing in Cloud Storage feeds a batch load. Networking and encryption are not afterthoughts you bolt on once the first request gets a 403; they decide whether the workers can even reach the source and sink, and whether the design meets the compliance bar. This subtopic walks the sketch in that order. The mechanics of writing the transform live in the sibling Building Pipelines subtopic, and the automation and deployment of the finished pipeline live in Deploying & Operationalizing; here the job is the design-time decision, not the build.
A worked read of the sketch for the fraud example: the source is a Pub/Sub stream of transaction events; the sink is Bigtable for low-latency lookups plus BigQuery for analysis; the latency contract is sub-second scoring, so the processor is Dataflow streaming; the network gives workers private IPs with Private Google Access so they call Pub/Sub and Bigtable without public addresses; and because the data is regulated, the keys are CMEK in Cloud KMS. Notice that no service was chosen until the requirement that selects it was written down.
Defining sources and sinks
A source is where raw data enters the pipeline; a sink is where the processed result must land for whoever consumes it. The planning rule is to match each one to its access pattern, because the wrong sink quietly caps the whole design no matter how good the processing is.
Sources by arrival shape
Sources fall into a few shapes the exam keeps reusing. Pub/Sub is the source for real-time events: it is a fully managed, serverless asynchronous messaging service that decouples publishers from subscribers, and in a streaming pipeline it is the ingestion layer that buffers and absorbs spikes in event volume, as covered in Pub/Sub overview[1]. Cloud Storage is the source for files and exports: it is object storage that holds raw data, the landing zone for batch loads, per the Cloud Storage docs[2]. Operational databases (Cloud SQL, Spanner, AlloyDB, or external) are sources when you must keep an analytics copy in sync; for that you plan a change-data-capture feed with Datastream[3], which lands changes in BigQuery or Cloud Storage. External and SaaS feeds (Google Ads, Salesforce, other warehouses) are sources for scheduled, no-code loads via BigQuery Data Transfer Service[4], whose destination is always BigQuery. The detailed selection of a migration mover is the Data Migrations subtopic; here you only need to identify the source's shape.
Sinks by access pattern
The sink is chosen by how the result will be read, and three cover most of the exam:
| Sink | Read pattern it serves | When to plan it |
|---|---|---|
| BigQuery[5] | SQL analytics over large scans | The result is queried, aggregated, and reported on; the default analytics sink |
| Bigtable[6] | Low-latency single-key lookups at high throughput | Apps need millisecond reads/writes by key, such as serving features or time-series |
| Cloud Storage[2] | Cheap durable object storage | Raw or archival output, a data-lake landing zone, or files for another system |
The trap that sinks designs is reaching for BigQuery when the consumer needs single-key millisecond reads: BigQuery is a warehouse built for scans, not a high-QPS key-value store, and Bigtable is the sink for that pattern. Conversely, do not pick Bigtable for ad-hoc SQL analytics; it has no SQL query engine of its own for that. A single pipeline often writes to more than one sink (Bigtable for serving plus BigQuery for analysis), and planning both up front is normal, not over-engineering.
Defining transformation and orchestration logic
Once the source, sink, and latency contract are fixed, two more planning choices remain: which service does the transformation, and what runs the multi-step workflow around it. These are different jobs and the exam tests them separately, so keep them apart from the start.
Batch versus streaming, then the processor
The first split is set by the latency contract from the planning sketch. Streaming processes each event within seconds of arrival and runs continuously; batch processes a bounded set on a schedule and is cheaper per unit of data. Choose streaming only when the freshness requirement demands it. With that axis set, the processing service is picked by team skills and where the data already lives:
- Dataflow is the serverless runner for Apache Beam[7] pipelines, with one unified model for stream and batch and autoscaling workers. Plan it for new custom transforms in code, especially streaming from Pub/Sub.
- Dataproc is managed open-source Apache Spark and Hadoop. Plan it to lift and shift existing OSS jobs[8] with minimal redevelopment; clusters are ephemeral and read from Cloud Storage, BigQuery, and Bigtable. The dividing line: existing Spark/Hadoop code goes to Dataproc, new Beam code to Dataflow.
- Cloud Data Fusion is a managed, code-free visual ETL service for analysts; under the hood it runs on ephemeral Dataproc clusters, so you pay only per run, per the Data Fusion overview[9].
- BigQuery SQL is the processor when the data already lives in BigQuery and the team is SQL-fluent; transform in place rather than moving data out to a separate engine.
The deep build mechanics of these services belong to Building Pipelines; the planning task here is the selection.
Orchestration: Composer versus Workflows
A processor runs one job; an orchestrator runs the ordered set of jobs that make a pipeline, with dependencies, retries, and a schedule. Plan the orchestrator as a separate decision, and the two candidates split cleanly:
Cloud Composer[10] is fully managed Apache Airflow. You author the workflow as a DAG (directed acyclic graph) of tasks in Python; Composer schedules them, respects dependencies, retries failures, and backfills. A Composer environment is an always-on Airflow deployment, so it carries a running cost whether or not a workflow is active. Plan Composer for scheduled, dependency-rich data pipelines that span Dataflow, BigQuery, Dataproc, and more.
Cloud Workflows[11] is a serverless orchestrator that chains service and API calls defined in YAML or JSON, and it is billed per step executed with no idle infrastructure. Plan Workflows for lightweight, event-triggered, or bursty orchestration where an always-on Airflow environment would be overkill. The exam line: heavy, scheduled, Python-defined data DAGs favor Composer; serverless API/service chaining with no idle cost favors Workflows.
The single sharpest distractor in this whole subtopic is letting the processor act as the orchestrator. Dataflow transforms data inside one job; it does not sequence a cross-service pipeline. When a question describes triggering a Dataflow job, then loading BigQuery, then refreshing a dashboard, the orchestrator is Composer (or Workflows), never Dataflow.
Planning networking and encryption
Networking and encryption are part of the design because a pipeline's workers, sources, and sinks all live inside a network and all carry data that must be protected. Plan both at design time; retrofitting them after a worker cannot reach an API or an auditor flags an unencrypted hop is expensive.
Network reach: private IPs and Private Google Access
The rule for pipeline workers is to keep them off the public internet and still let them reach Google APIs. By default a VM with no external IP cannot reach Google APIs over the internet, so you turn on Private Google Access on the subnet, which lets instances with only internal IP addresses reach Google APIs and services such as Pub/Sub, BigQuery, and Cloud Storage, per the Private Google Access docs[12]. Concretely for Dataflow[13], plan to launch workers with the option that gives them internal IPs only, on a subnet with Private Google Access enabled, so the job runs without any public addresses. Recall the network scope from the design layer: a VPC network and its firewall rules are global, while each subnet is regional, so the Dataflow job's subnet must be in the job's region.
A second, finer reach question is reaching a resource over an internal IP without VPC peering: Private Service Connect lets a consumer reach Google APIs or a published service over an internal IP inside its own VPC, scoped to a single service endpoint rather than a whole network. Plan it when one specific managed service must be reachable privately; plan plain Private Google Access when the goal is general access to Google APIs from internal-only workers.
The perimeter: VPC Service Controls
Network firewalls control packet-level VM traffic; they do not stop a caller with valid credentials from copying data out to a personal project. That exfiltration risk is what a VPC Service Controls perimeter addresses: it draws a boundary around projects and their Google-managed services, allowing free communication inside and blocking communication across the boundary by default, independent of IAM, per the VPC Service Controls overview[14]. For a pipeline handling regulated data, plan a perimeter around the projects holding the source, the processor, and the sink so that even a credentialed insider cannot move data to an outside project; controlled ingress and egress rules carve the exceptions, and dry-run mode logs would-be blocks before you enforce. Distinguish the two clearly: VPC firewall rules govern VM packets, VPC Service Controls governs which Google API calls may cross the perimeter.
Encryption: default first, CMEK when a rule requires it
Data in a Google Cloud pipeline is encrypted in transit and at rest automatically. At-rest encryption uses AES-256 with Google-managed keys by default, with no configuration and no cost. Plan customer-managed encryption keys (CMEK) in Cloud KMS[15] only when a compliance requirement says you must own, rotate, disable, or destroy the key; CMEK is the key-encryption key in envelope encryption, and disabling or destroying it makes the protected data unreadable (crypto-shredding). When you do plan CMEK, apply it consistently across the pipeline's resources (the Pub/Sub topic, the Dataflow job's temporary storage, the BigQuery dataset, the Cloud Storage bucket), because a single default-key resource in the chain undercuts the control. The deep mechanics of keys, CSEK, Cloud HSM, and EKM are the Security & Compliance subtopic; the planning task here is deciding default versus CMEK and applying the choice end to end.
Choosing the processing service at planning time
| Decision driver | Dataflow | Dataproc | Cloud Data Fusion | BigQuery SQL |
|---|---|---|---|---|
| Programming model | Apache Beam code (Java/Python) | Apache Spark/Hadoop code | Visual, code-free (CDAP) | SQL in the warehouse |
| Batch and streaming | Both, one unified model | Both (Spark batch + streaming) | Both, via pipeline type | Batch query; streaming via inserts |
| Best fit | New custom stream/batch transforms | Lift-and-shift existing OSS jobs | Analysts building ETL without code | Data already in BigQuery |
| Serverless | Yes, autoscaling workers | No, you size clusters (ephemeral) | Managed, runs on Dataproc under the hood | Yes, fully serverless |
Decision tree
Sharp facts the exam loves — give these one last read before exam day.
Cheat sheet
Sharp facts the exam loves — scan these before test day.
- Plan the source, sink, and latency contract before choosing any tool
Pipeline planning starts by naming three things on paper: the source (where data enters), the sink (where the result must land for consumers), and the latency contract (how fresh the result has to be). The latency requirement is the hinge that decides batch versus streaming, and the source and sink pin the ingestion and storage edges, so the processing service falls out of the requirements rather than driving them. Skipping this and picking a favorite tool first is how a design ends up with the wrong sink that caps it later.
- Let the latency contract, not preference, decide batch versus streaming
Choose streaming only when the freshness requirement is genuinely sub-second to seconds, because a streaming pipeline runs continuously and costs more to operate. Batch processes a bounded set on a schedule and is cheaper per unit of data, so a next-morning or weekly contract is a batch job. The deciding question is always how fresh the consumer needs the result, not how real-time the source happens to be.
Trap Reaching for streaming because the source emits events in real time, when the consumer only needs next-morning freshness and a cheaper batch job meets the contract.
- Pub/Sub is the source for real-time event ingestion
Pub/Sub is a fully managed, serverless asynchronous messaging service that decouples publishers from subscribers, and in a streaming pipeline it is the ingestion layer that buffers and absorbs spikes in event volume. Plan it as the source whenever events must be processed within seconds of arrival, feeding a streaming processor such as Dataflow. It is the standard front door of the canonical streaming pipeline: Pub/Sub ingests, Dataflow processes, BigQuery stores.
- Cloud Storage is the source for files and the landing zone for batch loads
Cloud Storage is object storage that holds raw data, so plan it as the source when the input is files, exports, or scheduled drops that a batch job will read. It doubles as a durable, cheap data-lake landing zone where raw output can sit before or after processing. Reach for it when the arrival shape is a file rather than a continuous event stream.
- Pick the sink by the consumer's read pattern, not by habit
The sink is chosen by how the result will be read: BigQuery for SQL analytics over large scans, Bigtable for low-latency single-key lookups at high throughput, and Cloud Storage for cheap durable archival or a data-lake landing zone. A pipeline often writes to more than one sink at once, such as Bigtable for serving and BigQuery for analysis. Matching the sink to the access pattern up front is the planning decision that keeps the design from being capped later.
- Use Bigtable, not BigQuery, when the consumer needs millisecond key lookups
Bigtable is a NoSQL wide-column store built for low-latency single-key reads and writes at high throughput, so it is the sink when an application serves data by key, such as feature serving or time-series lookups. BigQuery is a serverless analytics warehouse optimized for large scans and aggregation, not a high-QPS key-value store. Plan Bigtable for the serving path and BigQuery for the analytics path; many pipelines write both.
Trap Choosing BigQuery as the sink for an app needing single-key millisecond reads; it is a scan-oriented warehouse, so per-key high-QPS lookups belong in Bigtable.
- Plan Dataflow for new custom stream or batch transforms in code
Dataflow is the serverless, autoscaling runner for Apache Beam pipelines, with one unified programming model for both stream and batch. Plan it when the transformation is new custom code (Java or Python), especially streaming from Pub/Sub, where you want the same pipeline to handle bounded and unbounded data. It is the default processing stage of a GCP streaming pipeline.
Trap Building separate batch and streaming pipelines for the same logic on Dataflow; Apache Beam's unified model expresses both in one pipeline, so the second build is wasted effort.
- Plan Dataproc to lift and shift existing Spark or Hadoop jobs
Dataproc is managed open-source Apache Spark, Hadoop, Hive, and Presto/Trino with the same tools and APIs as on-premises, so existing OSS jobs move with minimal redevelopment and clusters spin up in about 90 seconds. The planning line versus Dataflow is the code you already have: existing Spark/Hadoop goes to Dataproc, new Apache Beam pipelines go to Dataflow, because Dataflow is a different programming model. Dataproc decouples storage from compute by reading Cloud Storage, BigQuery, and Bigtable, so clusters can be ephemeral.
Trap Rewriting working Spark or Hadoop code into Apache Beam for Dataflow when Dataproc would run the existing jobs as-is with far less migration effort.
- Plan Cloud Data Fusion when the builders are analysts, not engineers
Cloud Data Fusion is a managed, code-free visual ETL service built on CDAP, where analysts assemble pipelines by drag-and-drop instead of writing code. Plan it when the people building pipelines do not write Spark or Beam but need connectors and transforms. Under the hood it runs each pipeline on ephemeral Dataproc clusters it provisions and deletes per run, so you pay only for the run.
Trap Assuming Cloud Data Fusion has no compute cost because it is managed and code-free; it executes pipelines on ephemeral Dataproc clusters you pay for per run.
- Plan BigQuery SQL as the processor when the data already lives in BigQuery
When the data is already in BigQuery and the team is SQL-fluent, the cheapest, simplest processor is BigQuery itself: transform in place with SQL rather than exporting data to a separate engine. Plan this for in-warehouse ELT where moving data out to Dataflow or Dataproc would add cost and latency for no benefit. The processor choice is driven by where the data sits and the team's skills, not by defaulting to a dedicated processing service.
Trap Exporting data already in BigQuery out to Dataflow or Dataproc to transform it; an in-warehouse SQL transform avoids the data movement, extra cost, and added latency.
- Orchestration is a separate planning decision from processing
A single processing job is not a pipeline; a pipeline is several steps run in order with dependencies, retries, and a schedule, which is orchestration. Plan the orchestrator as its own decision rather than assuming the processor handles sequencing. Dataflow transforms data inside one job, but it does not sequence a cross-service workflow, so the multi-step pipeline needs Cloud Composer or Cloud Workflows around it.
Trap Treating Dataflow as the orchestrator of a cross-service pipeline; it processes data within one job, so sequencing a Dataflow job then a BigQuery load then a dashboard refresh needs Composer or Workflows.
- Choose Cloud Composer for scheduled, dependency-rich data DAGs
Cloud Composer is fully managed Apache Airflow: you author a workflow as a DAG of tasks in Python, and Composer schedules them, respects dependencies, retries failures, and backfills. Plan it for scheduled data pipelines that span several services such as Dataflow, BigQuery, and Dataproc. A Composer environment is an always-on Airflow deployment, so it carries a running cost whether or not a workflow is currently active.
- Choose Cloud Workflows for serverless, pay-per-step API chaining
Cloud Workflows is a serverless orchestrator that chains service and API calls defined in YAML or JSON, billed per step executed with no idle infrastructure. Plan it for lightweight, event-triggered, or bursty orchestration where an always-on Airflow environment would be overkill and you want to pay only when steps run. The exam split is clear: heavy scheduled Python-defined data DAGs favor Composer, serverless API and service chaining with no idle cost favors Workflows.
Trap Standing up a full Cloud Composer environment for a light, occasional sequence of API calls; the always-on Airflow cost is wasted when Workflows runs the same chain serverless and pay-per-step.
- Datastream is the source plan for keeping a BigQuery copy in sync with an operational database
When an operational database (Cloud SQL, PostgreSQL, MySQL, Oracle, SQL Server, AlloyDB, Spanner) must feed analytics in near real time, plan Datastream, a serverless change data capture (CDC) and replication service that streams changes with minimal latency into BigQuery or Cloud Storage. It keeps an analytics copy current without dumping and reloading the whole table. Choosing the full migration mover for a one-time move is the Data Migrations subtopic; here Datastream is the ongoing CDC source for a pipeline.
Trap Using a one-time bulk migration mover to keep a BigQuery copy current with an operational database; ongoing low-latency sync needs Datastream's change data capture, not a repeated full load.
- Give Dataflow workers internal IPs and turn on Private Google Access
Plan pipeline workers to run without external IP addresses, then enable Private Google Access on their subnet so instances with only internal IPs can still reach Google APIs and services such as Pub/Sub, BigQuery, and Cloud Storage. For Dataflow, launch workers with the internal-IP-only option on a Private Google Access subnet so the job runs entirely off the public internet. Without Private Google Access, an internal-only VM cannot reach Google APIs at all.
Trap Giving Dataflow workers internal IPs but leaving Private Google Access off the subnet; the workers then cannot reach Pub/Sub or BigQuery and the job fails to start.
- A VPC network is global, but a subnet (and the Dataflow job's subnet) is regional
A VPC network and its firewall rules are global resources not tied to any region, while each subnet is regional. When planning where a pipeline runs, the Dataflow job's worker subnet must be in the job's region, so regional placement of the subnet is a real design constraint even though the VPC itself spans every region. Reaching a different VPC needs VPC Network Peering, and on-premises needs Cloud VPN or Cloud Interconnect.
- Use a VPC Service Controls perimeter to stop data exfiltration that firewalls cannot
VPC firewall rules govern packet-level VM traffic, but they do not stop a caller with valid credentials from copying data out to a personal project. Plan a VPC Service Controls perimeter around the projects holding the source, processor, and sink: it blocks Google API calls crossing the boundary by default, independent of IAM, so even a credentialed insider cannot move regulated data outside. Controlled ingress and egress rules carve exceptions, and dry-run mode logs would-be blocks before you enforce.
Trap Relying on VPC firewall rules to prevent data exfiltration; firewalls filter VM packets, not which Google API calls cross a boundary, so the perimeter is VPC Service Controls.
- Plan default encryption first; add CMEK only when a rule requires key ownership
Pipeline data is encrypted in transit and at rest automatically, using AES-256 with Google-managed keys at rest, with no configuration and no cost. Plan customer-managed encryption keys (CMEK) in Cloud KMS only when a compliance requirement says you must own, rotate, disable, or destroy the key; disabling or destroying a CMEK makes the protected data unreadable (crypto-shredding). Adding CMEK where no rule demands it just buys the key-rotation burden for no benefit.
- Apply CMEK consistently across every resource in the pipeline chain
When the plan calls for CMEK, apply it to every resource the data passes through: the Pub/Sub topic, the Dataflow job's temporary storage, the BigQuery dataset, and the Cloud Storage bucket. A single resource left on the default Google-managed key undercuts the control, because the data sits unprotected by the customer key at that hop. The planning rule is end-to-end key coverage, not just the final sink.
Trap Setting CMEK on the BigQuery sink but leaving the Pub/Sub topic and Dataflow temp storage on default keys; the data is then outside the customer key for part of the pipeline.
References
- Pub/Sub overview
- Cloud Storage introduction
- Datastream overview
- BigQuery Data Transfer Service introduction
- BigQuery introduction
- Bigtable overview
- Dataflow overview
- Dataproc concepts overview
- Cloud Data Fusion overview
- Cloud Composer overview
- Cloud Workflows overview
- Private Google Access
- Dataflow networking: routes and firewall rules
- VPC Service Controls overview
- Customer-managed encryption keys (CMEK)