PDE Cheat Sheet

Cloud IAM grants a role (a bundle of permissions) to a principal on a resource; you never grant a bare permission. For least privilege, bind the narrowest role at the lowest level: an analyst who needs to read one BigQuery dataset gets roles/bigquery.dataViewer on that dataset, not roles/editor on the project. Both BigQuery datasets and tables and Cloud Storage buckets expose their own resource-level IAM, so a pipeline's service account can be scoped to exactly the data it touches.

Trap Granting a broad project-level role like Editor to satisfy a request that only needs read access on a single dataset, which silently exposes every other resource in the project.

An IAM allow policy set on an organization, folder, or project is inherited by every resource beneath it, so the level you bind a role at, not just the role you pick, decides how far it reaches. A role on the production folder applies to every project under it. This is why putting development and production in separate folders/projects is a real security control: a grant scoped to one environment cannot touch the other.

Trap Granting a role at the organization or folder to fix one project's access, which silently propagates the permission to every project below that node.

Allow policies are additive only and can never subtract a permission, so the way to guarantee a block is an IAM deny policy attached at organization, folder, or project. IAM evaluates deny rules before allow rules, so a matching deny blocks the permission regardless of any role granted, including an inherited Owner. A deny applies when its condition is true or cannot be evaluated, and exceptionPrincipals carves out break-glass identities.

Trap Trying to revoke an over-broad permission by adding more allow grants; allow is additive and never removes the original, so only a deny policy actually blocks it.

Cloud Storage can control access either with bucket-level uniform IAM or with legacy per-object ACLs, but mixing them makes access unauditable. For a bucket backing a data lake, enable uniform bucket-level access so all access is governed by IAM at the bucket, which is consistent with how datasets are governed and far easier to reason about than thousands of per-object ACLs.

Trap Leaving per-object ACLs enabled on a data-lake bucket, which scatters access decisions across objects and defeats a single auditable IAM policy.

Google Cloud encrypts all customer data at rest automatically with AES-256 using envelope encryption (a data encryption key wrapped by a key encryption key), with no action required. Encryption is therefore never the design decision; key ownership is. The question on the exam is whether Google-managed keys suffice or a regulator requires customer-controlled keys.

Customer-managed encryption keys (CMEK) put the key encryption key under your control in Cloud KMS, so you can rotate it on a schedule, disable it, and destroy it. Destroying or disabling the CMEK makes the protected data permanently unreadable, which is crypto-shredding. Choose CMEK when a regulator demands auditable key control or rotation; default Google-managed keys are simpler and add no Cloud KMS availability dependency when no such requirement exists.

Trap Adopting CMEK purely to be more secure when no requirement exists; data is already AES-256 encrypted, and CMEK only adds key lifecycle work and a dependency on Cloud KMS availability.

Both let the customer control the key, but they differ in storage. CMEK keys are created and managed inside Cloud KMS (Google stores the wrapped key). Customer-supplied encryption keys (CSEK) are raw keys passed with each request that Google uses then discards and never stores, so losing a CSEK loses the data with no recovery. CSEK shifts all key custody to you; CMEK keeps Cloud KMS lifecycle features.

Trap Assuming Google can recover data protected with CSEK; the key is never stored, so a lost CSEK is unrecoverable, unlike a CMEK held in Cloud KMS.

When a control mandates a hardware-backed key, Cloud HSM stores the CMEK in a cluster of FIPS 140-2 Level 3 certified hardware security modules. When the key must live entirely outside Google Cloud, Cloud EKM (External Key Manager) keeps it in a third-party key manager that Cloud KMS calls out to. Plain CMEK is a software-protected key in Cloud KMS, the default tier when neither hardware certification nor external custody is required.

Trap Reaching for Cloud EKM when the requirement is only a hardware-backed key; FIPS 140-2 Level 3 is satisfied by Cloud HSM inside Google Cloud, and EKM is for keeping the key outside Google entirely.

Rather than configuring a key per table, set a default CMEK on the BigQuery dataset and new tables created in it inherit that key. In Cloud Storage the equivalent is a default CMEK on the bucket. This makes key control a one-time dataset/bucket configuration instead of a per-object task that someone will forget.

Sensitive Data Protection (formerly Cloud DLP) scans BigQuery, Cloud Storage, and Datastore, and can inspect streaming content, matching built-in infoTypes such as credit-card number or national ID. It then de-identifies findings by masking, redaction, bucketing (generalizing a value, for example an exact age to a range), date shifting, or tokenization. Reach for it when sensitive data spans services or needs a bulk de-identification job, because the protection acts on the values themselves and survives an export.

Trap Relying on IAM alone to protect PII; IAM only controls who opens the table, but the raw values remain exposed once they appear in a query result or export.

Tokenization (pseudonymization) in Sensitive Data Protection replaces a sensitive value with a token while preserving a consistent mapping, so the same input always produces the same token and records can still be joined or grouped by it. Unlike redaction or masking, the original can be recovered when the cryptographic or format-preserving method allows re-identification by an authorized party. Use it when analysts must work with the data but never see the real identifiers.

Trap Treating tokenization as the same as masking or redaction; those destroy the value, while tokenization keeps a reversible, consistent mapping for re-identification.

BigQuery column-level security tags sensitive columns with policy tags from Data Catalog and grants the Fine-Grained Reader role only to principals allowed to read them; everyone else simply cannot select the tagged column. This restricts access per column without copying the table or building separate views, keeping one governed source of truth in the warehouse.

Dynamic data masking builds on policy tags to return an obfuscated value at query time based on the caller role, so the same query shows real data to a privileged role and masked data to others, with no second copy of the table. A caller with the BigQuery Masked Reader role sees the masked value while a Fine-Grained Reader sees the real one, and the masking rule can nullify the column, return a default for its type, or hash it (SHA-256). Choose masking over plain column-level security when lower-privileged users still need to run the query but must not see the real values.

Trap Creating duplicate masked copies or separate views per audience to hide columns, when dynamic data masking returns role-appropriate values from the single table at query time.

You choose a dataset's location (a region or multi-region) when you create it, and it cannot be changed afterward; relocating data means creating a new dataset in the target location and copying. Because all tables in a dataset inherit its location, dataset location is the lever for data residency, and queries cannot join tables across locations.

Trap Assuming a dataset can be moved to another region after creation; the location is immutable, so residency must be decided up front or the dataset recreated and copied.

Setting locations per dataset is not enough at scale because a team can still create resources in the wrong region. The Organization Policy resource-location constraint, set at the organization or folder, forces every project below to provision resources only in allowed regions, turning residency from advisory into enforced. Organization Policy governs what configuration is allowed (the what) while IAM governs who can act (the who); a constraint inherits to all descendants of the node it is set on.

Trap Trusting each team to pick the right region instead of setting a resource-location org policy; without the constraint nothing prevents provisioning outside the required jurisdiction.

When a requirement is residency alone, a resource-location Organization Policy is sufficient and far lighter. Assured Workloads is for scenarios that need location AND access AND personnel guarantees together, such as an EU Data Boundary that also restricts which Google support staff may act, enforced through control packages tied to specific compliance regimes. Reach for it only when those combined guarantees are stated, not for plain residency.

Trap Reaching for Assured Workloads when only data residency is needed; a resource-location org policy already pins the region, and Assured Workloads adds heavier access and personnel controls you were not asked for.

A VPC Service Controls service perimeter surrounds projects and their Google-managed services: communication inside is free and communication across the boundary is blocked by default. It is context-based and independent of IAM, so a caller holding valid IAM permissions is still denied across the perimeter, which makes it the data-exfiltration control. Controlled exceptions are ingress and egress rules, and dry-run mode logs would-be blocks before enforcing.

Trap Expecting VPC Service Controls to filter packet-level VM traffic; it governs Google API access such as a BigQuery API call, while VPC firewall rules handle network packets between VMs.

Putting development and production in separate Google Cloud projects ensures a test job can never read or corrupt production data and that IAM, quotas, billing, and audit logs are scoped cleanly per environment. A folder per environment lets one inherited IAM and Organization Policy set govern every project beneath it, so controls and isolation are applied once at the folder rather than reconfigured per project.

Trap Sharing one project across dev and prod to save setup, which lets test workloads reach production data and entangles billing, quotas, and audit logs across environments.

Because dataset-level grants are the natural BigQuery IAM boundary, group tables into datasets along the lines teams actually share data, so access is granted by dataset rather than clawed back table by table. A dataset per team or per sensitivity tier keeps the grant simple and auditable, whereas one giant dataset forces fine-grained exceptions that drift over time.

Enforce constraints/iam.disableServiceAccountKeyCreation at the organization level to stop anyone from creating long-lived service-account keys, regardless of their IAM permissions; pair it with iam.automaticIamGrantsForDefaultServiceAccounts to stop default service accounts getting Editor automatically. Both are enforced by default for organizations created after May 3, 2024. Grant emergency exceptions by overriding the constraint with enforcement disabled on a specific project, not by loosening IAM.

Trap Reaching for IAM role tweaks or written policy to stop key creation when only an organization policy constraint can block it across all current and future projects.

To protect BigQuery (or Cloud Storage) with a CMEK key, grant that product's service account roles/cloudkms.cryptoKeyEncrypterDecrypter on the Cloud KMS key before creating the resource. Hosting the key in a separate, dedicated KMS project gives separation of duties so key administrators are not the same principals that use the key in workloads.

Trap Granting the human user or workload identity the encrypt/decrypt role instead of the product's own service agent, which is the principal BigQuery actually uses to call KMS.

Cloud KMS requires the key ring's location to match the data's location: a US multi-region BigQuery dataset or bucket needs a key ring in the US multi-region, and a us-east1 bucket needs a key ring in us-east1. A multi-region resource cannot use a regional key, and a regional resource cannot borrow a key from another region.

Trap Creating one regional key ring (e.g. us-central1) and trying to reuse it for buckets or datasets in other regions or multi-regions.

Key rotation only creates a new primary version for future writes. Existing BigQuery tables keep the key version they were created with, and existing Cloud Storage objects keep their original version; only newly created tables/objects use the new version. To move existing data onto the latest version you must explicitly re-encrypt it (e.g. bq update with --destination_kms_key, or copying the data with the new key specified).

Trap Assuming automatic rotation re-encrypts old tables and satisfies a compliance requirement for current-version encryption without any further action.

Add a service such as bigquery.googleapis.com to the Deny list of constraints/gcp.restrictNonCmekServices and any resource created in that service without a CMEK key is rejected, while existing resources stay accessible. Combine it with constraints/gcp.restrictCmekCryptoKeyProjects to require that the keys come only from an approved KMS project.

Trap Trying to force CMEK with IAM permissions or per-table settings instead of the organization policy constraint that fails non-CMEK resource creation.

Unlike most services where Data Access audit logs are off until you enable them, BigQuery's Data Access logs are always on and immutable, giving immediate who-accessed-what-and-when visibility. For org-wide compliance, route them with an organization-level aggregated sink using --include-children so current and future projects are covered, and remember Admin Activity and System Event logs live in the _Required bucket with a fixed 400-day retention.

Trap Enabling Data Access logs for BigQuery from scratch (or assuming they are off) when they are already capturing access by default.

VPC Service Controls ingress rules let outside principals (e.g. a partner project/service account) reach protected resources, scoped down to specific identities and read-only methods; access levels in Access Context Manager add IP-range and device-trust conditions. For maximum exfiltration protection with least overhead, prefer one common unified perimeter over many small perimeters, and diagnose blocks with the VPC-SC troubleshooter using the unique ID in the denial.

Trap Using an egress rule (outbound) when the requirement is to let an external partner read in, which is what an ingress rule controls.

Grant roles/iam.serviceAccountUser on a service account so a developer can attach it to a job (like Dataflow) without being able to create, delete, or modify the account. Grant roles/iam.serviceAccountTokenCreator when someone must impersonate the account to generate short-lived credentials, for example to test locally as a production pipeline's identity without holding standing access to production data.

Trap Granting Token Creator when the user only needs to run a job under a service account, or handing out a downloaded service-account key instead of impersonation.

Treat the two as separate properties a design must satisfy independently. Reliability is operational: the pipeline keeps delivering through zone and region failures, restarts cleanly, and is monitored. Fidelity is data correctness: records are cleaned, validated, and never silently corrupted or lost. A pipeline can run flawlessly and still publish wrong numbers, so a scenario that asks about trustworthy data is usually testing the fidelity half even when everything is up.

When cleaning logic is complex, real-time, or needs custom code, use Dataflow, the serverless managed runner for Apache Beam pipelines that handles stream and batch with one model and autoscaling workers. It is the standard processing stage of a streaming pipeline (Pub/Sub ingests, Dataflow transforms, BigQuery stores). Reach for Cloud Data Fusion instead when the builders are analysts who want a visual, code-free tool, or Dataform when the data is already in BigQuery and the team writes SQL.

Trap Reaching for Cloud Composer to do the transformation; Composer orchestrates the steps but does not process the data, Dataflow does.

When the people building pipelines are data analysts rather than engineers, use Cloud Data Fusion, a fully managed data integration service with a drag-and-drop interface built on the open-source CDAP project. Its Wrangler tool prepares and cleans data with point-and-click directives, and it runs pipelines on ephemeral Dataproc (Spark) clusters it provisions and deletes per run. Choose Dataflow instead when you need custom code or low-latency streaming logic.

When data is already loaded in BigQuery and the team is SQL-fluent, use Dataform to run the transform step of ELT in place. Developers write SQLX (SQL plus metadata) files and the ref() function builds the dependency DAG automatically, so you transform without moving data to a separate engine. This is the ELT pattern (transform after load), distinct from Dataflow's ETL where transformation happens in flight before the warehouse.

An assertion in Dataform is a data-quality test written as SQL that queries for rows violating a condition; if any violating rows return, the assertion fails. They check uniqueness, null values, or a custom condition, and because they are nodes in the same dependency DAG as the transforms, you can require a table's assertions to pass before downstream tables run, stopping corruption from propagating. Put the check in the pipeline so a bad batch fails closed rather than reaching consumers.

Trap Validating data only by inspecting the output dashboard afterward; that catches bad data after consumers have already read it, not before it lands.

Use Dataplex auto data quality when you want a managed, scheduled scan that validates a BigQuery table and alerts on failure. Rules are row-level (per-row, with a passing-threshold percentage: range, null, regex) or aggregate (across the dataset: uniqueness, statistic), and each carries one of seven dimensions: freshness, volume, completeness, validity, consistency, accuracy, uniqueness. A scan produces a data-quality score (percentage of rules passed); the standard pattern is to gate the pipeline by failing it when the score drops below target.

Separate the two validation layers. Schema validation rejects records whose shape is wrong (missing required fields, wrong types) and is enforced by BigQuery on load; value rules (assertions, Dataplex rules) check the content of structurally-valid rows. A Dataflow pipeline routes records that fail schema parsing to a dead-letter destination so one malformed row does not crash the whole job. Use both: schema is the structural floor, assertions are the content gate.

Trap Treating a passed schema check as proof the data is correct; schema validation only confirms shape, not that the values are valid or complete.

When a streaming Dataflow job hits records it cannot parse or that fail validation, send them to a dead-letter destination (a separate Pub/Sub topic, BigQuery table, or Cloud Storage path) rather than letting one bad record crash the pipeline. The good data keeps flowing and the rejected records are captured for later inspection and reprocessing. This preserves both reliability (the job stays up) and fidelity (bad rows never silently enter the clean dataset).

When a pipeline spans several services with dependencies, retries, and a schedule, orchestrate it with Cloud Composer, the managed Apache Airflow service where you declare the workflow as a DAG of tasks in Python. Composer schedules tasks, respects dependencies, and retries failures; a Composer environment runs Airflow components on a GKE cluster that Composer manages. Do not confuse it with Dataflow, which processes the data inside one job and may itself be a single task in a Composer DAG.

Trap Naming Dataflow as the orchestrator of a cross-service pipeline; Dataflow runs one processing job, Composer sequences the whole multi-step workflow.

Monitor a pipeline with Cloud Monitoring alerting policies, which bundle a condition (metric, threshold, aggregation) with notification channels and open an incident when met. Dataflow exposes job metrics such as system lag and data freshness that you alert on directly. Anchor alerts to user-felt symptoms (rising lag, falling freshness, climbing error rate), not to every internal CPU spike, so on-call gets signal rather than noise.

When the failure you care about appears in logs but no built-in metric exposes it, create a Cloud Logging log-based metric that turns matching log entries into a Cloud Monitoring time series you can chart and alert on. A counter metric counts matching entries; a distribution metric extracts a number from each entry and builds a histogram. This is how you alert on error strings or parsed values that the standard Dataflow or service metrics do not surface.

ACID describes what a transactional store guarantees for each transaction: atomicity (all-or-nothing), consistency (only valid state transitions), isolation (concurrent transactions do not corrupt each other), and durability (a committed write survives a crash). It is the property you match a database choice against, so a scenario demanding that a multi-row financial update never partially apply is asking for an ACID-compliant transactional store, not an analytics warehouse.

When a relational workload needs both horizontal global scale and strong consistency, choose Spanner, which provides ACID transactions with external consistency, the strictest concurrency guarantee, backed by the TrueTime synchronized clock, and carries up to a 99.999% availability SLA. External consistency means the system behaves as if all transactions ran one at a time in commit order, even across regions. Use Cloud SQL instead when a single region's relational workload is enough; Spanner's cost is justified only when global scale and strong consistency are both required.

Trap Reaching for Spanner for a single-region transactional app that Cloud SQL handles; you pay for global scale you do not need.

Match the store to the consistency and access pattern. Cloud SQL (managed MySQL, PostgreSQL, SQL Server) gives full ACID for one regional transactional application. BigQuery is a serverless analytics warehouse for reporting, not a high-QPS transactional system, so it is the wrong answer for transaction processing (OLTP) even though it speaks SQL. Bigtable is NoSQL wide-column with single-row atomic operations only and no multi-row ACID, suited to high-throughput single-key access such as time-series or IoT data.

Trap Using BigQuery as the transactional database because it runs SQL; it is an analytics warehouse, not an OLTP store for frequent small transactions.

When stopping a running Dataflow streaming job, drain stops reading new input but lets the service finish processing buffered, in-flight data before shutting down, so no records are lost; the trade-off is that draining can take significant time. Cancel stops all processing immediately, including buffered data, so it is fast but you may lose in-flight data. To redeploy a new pipeline version without losing records, drain; cancel only when data loss is acceptable and you need an immediate stop.

Trap Cancelling a streaming job to deploy a new version; cancel discards buffered and in-flight data, so use drain to preserve it during a redeploy.

RTO (recovery time objective) is the maximum tolerable downtime and RPO (recovery point objective) is the maximum tolerable data loss; both drive the DR design, and tighter values cost more. Set backup frequency to meet the RPO (a 15-minute RPO means backing up at least every 15 minutes), and match readiness posture to the RTO: cold (backup-and-restore, cheapest, slowest), warm (scaled-down standby), or hot (full second deployment, fastest, priciest). Buy the strictest posture only when the business requirement demands it.

Trap Buying an always-on multi-region hot standby for a workload whose RTO/RPO budget a cheaper backup-and-restore posture already meets.

Make a stopped pipeline resumable rather than restarting from scratch. Keep events in Pub/Sub so a failed downstream pipeline can replay unacknowledged messages and resume without a gap, and checkpoint long-running jobs so a restart resumes from the last consistent point. These recover within the RPO by not losing the records that were in flight when the failure hit. Pub/Sub retains undelivered messages, which is what makes the replay possible.

Testing recovery is part of reliability, not an afterthought. The reliability discipline requires regularly restoring backups in a non-production environment to confirm they contain consistent, usable snapshots, because a backup that has never been restored is not yet proven recoverable. Schedule recovery drills (for example, periodic regional-failover and restore-from-backup tests) rather than discovering a broken backup during the real outage.

Trap Assuming a successful backup job means the data is recoverable; only a completed test restore proves the snapshot is consistent and usable.

Generative AI such as Gemini can draft the SQL for a Dataform or BigQuery transformation from a natural-language description, which speeds authoring. The generated query is a draft, not a verified artifact, so you review it and run it through the same assertion and data-quality gates as hand-written SQL before it touches production. The LLM accelerates writing the transform; it does not replace validating its output.

Time travel only recovers data for 7 days by default, so for longer retention (14, 30, 365 days) automate CREATE SNAPSHOT TABLE DDL with a scheduled query and an expiration matching the requirement, writing snapshots to a separate dataset so they survive deletion of the source. Snapshots are cheap because BigQuery only charges for data that differs from the base table.

Trap Relying on time travel or copying full tables for long-term retention instead of expiring snapshots, which only bill for the delta from the base table.

For automated failover of both compute and storage across regions with a cross-regional SLA, use BigQuery managed disaster recovery with an Enterprise Plus edition failover reservation and attach the replicated datasets to it. A soft failover needs both regions available and replicates everything (no loss); a hard failover promotes the secondary immediately and can lose unreplicated data. This is separate from BigQuery's native single-region zonal redundancy.

Trap Assuming BigQuery's built-in zonal redundancy already covers a full regional outage; cross-region failover requires the Enterprise Plus managed-DR feature.

XCom is for small metadata. Pushing megabyte-scale objects through it bloats the Airflow metadata database, degrades scheduler performance, and breaks environment snapshots and upgrades. Store the actual dataset in Cloud Storage and pass just the GCS path/URI via XCom, letting the downstream task fetch it.

Trap Treating XCom as a general data-transfer channel and serializing the whole intermediate dataset into it instead of a reference.

Maintenance and worker restarts terminate and reschedule tasks, so design each task to do one operation that is safe to re-run, and configure retries (default_args / default_task_retries, with retry_exponential_backoff for flaky APIs). Achieve idempotent writes with BigQuery MERGE or WRITE_TRUNCATE keyed on the logical run date, and use Jinja macros like {{ ds }} / {{ execution_date }} rather than datetime.today() so historical reruns target the correct partition.

Trap Using datetime.today() or append-only inserts in a DAG, so a retry stamps the current date and produces duplicate rows on re-execution.

A dead-letter path should use the simplest, most reliable sink (often Cloud Storage rather than BigQuery) because a failure writing to the dead-letter sink can fail the whole pipeline. Capture the original input record plus the error at the earliest validation stage; branching later only saves a partially transformed element you can't reprocess. In Dataflow, route multiple outcomes with a ParDo and TupleTag side outputs, and note Pub/Sub dead-lettering only fires for failures in the first fused stage because Dataflow acks the message after that stage.

Trap Putting the dead-letter branch after transformations, so the captured record is partial and can't be matched back to the source for reprocessing.

Store Spark Structured Streaming checkpoints in Cloud Storage, with a unique checkpoint path per query, so progress survives cluster failures. Checkpoints and write-ahead logs alone are not enough for exactly-once delivery: the output sink must be idempotent, e.g. use foreachBatch with the batchId and MERGE/upsert on a business key so reprocessed records don't duplicate. For driver-failure resilience add an HA cluster and StreamingContext.getOrCreate.

Trap Believing checkpointing by itself gives exactly-once output; without an idempotent sink, recovery replays can write duplicates.

Supply a precomputed checksum on upload and Cloud Storage validates it server-side, rejecting the object with a 400 if it doesn't match so corrupt data is never stored. For parallel composite uploads and compose results, only CRC32C is available (MD5 is not), so validate each component and the final composite with CRC32C; Python clients need google-crc32c or crcmod to compute it efficiently.

Trap Expecting an MD5 hash on a composite object for integrity checks, when composite objects only carry a CRC32C.

When multiple writers update the same object, read its generation (content) and metageneration (metadata) numbers, then include generation-match / metageneration-match preconditions in the write; the request fails with 412 if another process changed it first, so no silent overwrite occurs. Pin the generation number in ranged reads too, so all chunks of a large object come from one consistent version.

Trap Relying on retries or object locking for concurrent writers instead of optimistic generation preconditions that make a stale write fail fast.

Enable Object Versioning to keep readable noncurrent versions of overwritten or deleted objects, and use Object Lifecycle Management to expire those noncurrent versions and cap storage cost. Object Versioning does not protect against deleting the bucket itself; soft delete does, letting you restore a deleted bucket and its objects. Enable both for layered protection against accidental and malicious deletion.

Trap Assuming Object Versioning protects against bucket deletion; only soft delete can restore a deleted bucket.

Pipeline planning starts by naming three things on paper: the source (where data enters), the sink (where the result must land for consumers), and the latency contract (how fresh the result has to be). The latency requirement is the hinge that decides batch versus streaming, and the source and sink pin the ingestion and storage edges, so the processing service falls out of the requirements rather than driving them. Skipping this and picking a favorite tool first is how a design ends up with the wrong sink that caps it later.

Choose streaming only when the freshness requirement is genuinely sub-second to seconds, because a streaming pipeline runs continuously and costs more to operate. Batch processes a bounded set on a schedule and is cheaper per unit of data, so a next-morning or weekly contract is a batch job. The deciding question is always how fresh the consumer needs the result, not how real-time the source happens to be.

Trap Reaching for streaming because the source emits events in real time, when the consumer only needs next-morning freshness and a cheaper batch job meets the contract.

Pub/Sub is a fully managed, serverless asynchronous messaging service that decouples publishers from subscribers, and in a streaming pipeline it is the ingestion layer that buffers and absorbs spikes in event volume. Plan it as the source whenever events must be processed within seconds of arrival, feeding a streaming processor such as Dataflow. It is the standard front door of the canonical streaming pipeline: Pub/Sub ingests, Dataflow processes, BigQuery stores.

Cloud Storage is object storage that holds raw data, so plan it as the source when the input is files, exports, or scheduled drops that a batch job will read. It doubles as a durable, cheap data-lake landing zone where raw output can sit before or after processing. Reach for it when the arrival shape is a file rather than a continuous event stream.

The sink is chosen by how the result will be read: BigQuery for SQL analytics over large scans, Bigtable for low-latency single-key lookups at high throughput, and Cloud Storage for cheap durable archival or a data-lake landing zone. A pipeline often writes to more than one sink at once, such as Bigtable for serving and BigQuery for analysis. Matching the sink to the access pattern up front is the planning decision that keeps the design from being capped later.

Bigtable is a NoSQL wide-column store built for low-latency single-key reads and writes at high throughput, so it is the sink when an application serves data by key, such as feature serving or time-series lookups. BigQuery is a serverless analytics warehouse optimized for large scans and aggregation, not a high-QPS key-value store. Plan Bigtable for the serving path and BigQuery for the analytics path; many pipelines write both.

Trap Choosing BigQuery as the sink for an app needing single-key millisecond reads; it is a scan-oriented warehouse, so per-key high-QPS lookups belong in Bigtable.

Dataflow is the serverless, autoscaling runner for Apache Beam pipelines, with one unified programming model for both stream and batch. Plan it when the transformation is new custom code (Java or Python), especially streaming from Pub/Sub, where you want the same pipeline to handle bounded and unbounded data. It is the default processing stage of a GCP streaming pipeline.

Trap Building separate batch and streaming pipelines for the same logic on Dataflow; Apache Beam's unified model expresses both in one pipeline, so the second build is wasted effort.

Dataproc is managed open-source Apache Spark, Hadoop, Hive, and Presto/Trino with the same tools and APIs as on-premises, so existing OSS jobs move with minimal redevelopment and clusters spin up in about 90 seconds. The planning line versus Dataflow is the code you already have: existing Spark/Hadoop goes to Dataproc, new Apache Beam pipelines go to Dataflow, because Dataflow is a different programming model. Dataproc decouples storage from compute by reading Cloud Storage, BigQuery, and Bigtable, so clusters can be ephemeral.

Trap Rewriting working Spark or Hadoop code into Apache Beam for Dataflow when Dataproc would run the existing jobs as-is with far less migration effort.

Cloud Data Fusion is a managed, code-free visual ETL service built on CDAP, where analysts assemble pipelines by drag-and-drop instead of writing code. Plan it when the people building pipelines do not write Spark or Beam but need connectors and transforms. Under the hood it runs each pipeline on ephemeral Dataproc clusters it provisions and deletes per run, so you pay only for the run.

Trap Assuming Cloud Data Fusion has no compute cost because it is managed and code-free; it executes pipelines on ephemeral Dataproc clusters you pay for per run.

When the data is already in BigQuery and the team is SQL-fluent, the cheapest, simplest processor is BigQuery itself: transform in place with SQL rather than exporting data to a separate engine. Plan this for in-warehouse ELT where moving data out to Dataflow or Dataproc would add cost and latency for no benefit. The processor choice is driven by where the data sits and the team's skills, not by defaulting to a dedicated processing service.

Trap Exporting data already in BigQuery out to Dataflow or Dataproc to transform it; an in-warehouse SQL transform avoids the data movement, extra cost, and added latency.

A single processing job is not a pipeline; a pipeline is several steps run in order with dependencies, retries, and a schedule, which is orchestration. Plan the orchestrator as its own decision rather than assuming the processor handles sequencing. Dataflow transforms data inside one job, but it does not sequence a cross-service workflow, so the multi-step pipeline needs Cloud Composer or Cloud Workflows around it.

Trap Treating Dataflow as the orchestrator of a cross-service pipeline; it processes data within one job, so sequencing a Dataflow job then a BigQuery load then a dashboard refresh needs Composer or Workflows.

Cloud Composer is fully managed Apache Airflow: you author a workflow as a DAG of tasks in Python, and Composer schedules them, respects dependencies, retries failures, and backfills. Plan it for scheduled data pipelines that span several services such as Dataflow, BigQuery, and Dataproc. A Composer environment is an always-on Airflow deployment, so it carries a running cost whether or not a workflow is currently active.

Cloud Workflows is a serverless orchestrator that chains service and API calls defined in YAML or JSON, billed per step executed with no idle infrastructure. Plan it for lightweight, event-triggered, or bursty orchestration where an always-on Airflow environment would be overkill and you want to pay only when steps run. The exam split is clear: heavy scheduled Python-defined data DAGs favor Composer, serverless API and service chaining with no idle cost favors Workflows.

Trap Standing up a full Cloud Composer environment for a light, occasional sequence of API calls; the always-on Airflow cost is wasted when Workflows runs the same chain serverless and pay-per-step.

When an operational database (Cloud SQL, PostgreSQL, MySQL, Oracle, SQL Server, AlloyDB, Spanner) must feed analytics in near real time, plan Datastream, a serverless change data capture (CDC) and replication service that streams changes with minimal latency into BigQuery or Cloud Storage. It keeps an analytics copy current without dumping and reloading the whole table. Choosing the full migration mover for a one-time move is the Data Migrations subtopic; here Datastream is the ongoing CDC source for a pipeline.

Trap Using a one-time bulk migration mover to keep a BigQuery copy current with an operational database; ongoing low-latency sync needs Datastream's change data capture, not a repeated full load.

Plan pipeline workers to run without external IP addresses, then enable Private Google Access on their subnet so instances with only internal IPs can still reach Google APIs and services such as Pub/Sub, BigQuery, and Cloud Storage. For Dataflow, launch workers with the internal-IP-only option on a Private Google Access subnet so the job runs entirely off the public internet. Without Private Google Access, an internal-only VM cannot reach Google APIs at all.

Trap Giving Dataflow workers internal IPs but leaving Private Google Access off the subnet; the workers then cannot reach Pub/Sub or BigQuery and the job fails to start.

A VPC network and its firewall rules are global resources not tied to any region, while each subnet is regional. When planning where a pipeline runs, the Dataflow job's worker subnet must be in the job's region, so regional placement of the subnet is a real design constraint even though the VPC itself spans every region. Reaching a different VPC needs VPC Network Peering, and on-premises needs Cloud VPN or Cloud Interconnect.

VPC firewall rules govern packet-level VM traffic, but they do not stop a caller with valid credentials from copying data out to a personal project. Plan a VPC Service Controls perimeter around the projects holding the source, processor, and sink: it blocks Google API calls crossing the boundary by default, independent of IAM, so even a credentialed insider cannot move regulated data outside. Controlled ingress and egress rules carve exceptions, and dry-run mode logs would-be blocks before you enforce.

Trap Relying on VPC firewall rules to prevent data exfiltration; firewalls filter VM packets, not which Google API calls cross a boundary, so the perimeter is VPC Service Controls.

Pipeline data is encrypted in transit and at rest automatically, using AES-256 with Google-managed keys at rest, with no configuration and no cost. Plan customer-managed encryption keys (CMEK) in Cloud KMS only when a compliance requirement says you must own, rotate, disable, or destroy the key; disabling or destroying a CMEK makes the protected data unreadable (crypto-shredding). Adding CMEK where no rule demands it just buys the key-rotation burden for no benefit.

When the plan calls for CMEK, apply it to every resource the data passes through: the Pub/Sub topic, the Dataflow job's temporary storage, the BigQuery dataset, and the Cloud Storage bucket. A single resource left on the default Google-managed key undercuts the control, because the data sits unprotected by the customer key at that hop. The planning rule is end-to-end key coverage, not just the final sink.

Trap Setting CMEK on the BigQuery sink but leaving the Pub/Sub topic and Dataflow temp storage on default keys; the data is then outside the customer key for part of the pipeline.

The canonical Google Cloud streaming pipeline pairs Pub/Sub as the real-time ingestion layer with Dataflow as the processing stage, writing the result to BigQuery. Pub/Sub is fully managed, serverless messaging that decouples publishers from subscribers and buffers spikes in event volume so the processor is never overrun; Dataflow then applies the transform and aggregation logic. When a stem describes events arriving continuously and needing windowed aggregation, this is the build.

Trap Naming Cloud Composer as the stream processor; Composer orchestrates multi-step workflows, it does not transform the data itself.

Dataflow runs Apache Beam pipelines and uses the same programming model for batch and stream analytics, so you write the transform logic once. The only difference is the input: batch processes a bounded dataset that is complete when the job starts, while streaming processes an unbounded dataset that never ends. Because the code is identical, you can run the same pipeline on a stream by swapping the source for Pub/Sub.

An Apache Beam pipeline defines what to compute, not how to execute it; a runner carries it out on a specific platform, and the Dataflow runner is the managed, serverless, autoscaling runner on Google Cloud. This separation is why the same Beam code can target different runners. Dataflow provisions and manages all the infrastructure, so there are no clusters to size.

Dataproc is managed open-source Apache Spark, Hadoop, Hive, and Presto/Trino with the same tools and APIs as on-premises, so existing OSS jobs lift and shift with minimal redevelopment. The decision against Dataflow is about code, not capability: pick Dataproc when migrating Spark or Hadoop jobs you already have, and Dataflow when authoring new pipelines in the Beam model. Dataproc clusters create and shut down in about 90 seconds and read from Cloud Storage, BigQuery, and Bigtable, so they can be ephemeral and billed per run.

Trap Rewriting working Spark/Hadoop jobs into Apache Beam for Dataflow when Dataproc would run them as-is with minimal change.

Cloud Data Fusion builds ETL pipelines through a drag-and-drop visual interface with no code, which fits when the builders are data analysts rather than software engineers. It includes Wrangler, an interactive tool that parses, filters, and reshapes data with point-and-click directives for cleansing. Under the hood it executes pipelines on ephemeral Dataproc clusters it provisions per run, so you pay only for the run.

Dataform manages the ELT pattern inside BigQuery: the data is already loaded and Dataform runs the transform step as version-controlled SQLX (SQL plus metadata), compiling the dependencies into a DAG (directed acyclic graph, the ordered task-dependency map) it runs in BigQuery. Choose it when the data is already in BigQuery and the team is SQL-fluent, so you transform in place instead of moving data to a separate engine. For one-off SQL transforms, plain BigQuery SQL serves the same role without the project structure.

A tumbling window, which Google also calls a fixed time window, is a consistent, disjoint time interval: back-to-back, non-overlapping buckets such as 0:00-0:30 then 0:30-1:00. Each element belongs to exactly one window, which makes tumbling the right choice for periodic, non-overlapping aggregations like per-minute counts. Use it when each event should be counted once and only once.

Trap Using a tumbling window for a moving average; disjoint windows never overlap, so a rolling metric needs a hopping (sliding) window instead.

A hopping window, also called a sliding time window, is a consistent interval that can overlap, defined by a window length plus a shorter period at which a new window starts. Because windows overlap, one element can belong to multiple windows, which is exactly what a moving average needs: a 5-minute window recomputed every 1 minute reports the last 5 minutes' value once per minute. The period controls how often you emit, the length controls how much history each emit covers.

A session window contains elements within a gap duration of one another and is keyed, so it groups a burst of activity and closes when a gap of inactivity passes without new data. The window boundaries come from the data, not the clock, which fits per-user clickstream sessions or activity bursts of unpredictable length. The gap duration, not a fixed size, is the tuning knob.

Trap Forcing a fixed-size window onto irregular per-user activity; session windows exist precisely because the boundaries depend on data gaps, not a clock interval.

Because streamed data is not guaranteed to arrive in time order or at predictable intervals, the runner needs an estimate of completeness. A watermark is a threshold indicating when Dataflow expects all of a window's data to have arrived; when the watermark passes the end of a window, the runner treats it as ready and, by default, emits its result. The watermark is about completeness, not about when you choose to emit, which is the trigger's job.

Trap Confusing the watermark with the trigger; the watermark estimates completeness, the trigger decides when to emit results.

A trigger determines when to emit aggregated results as data arrives. The default trigger fires once, when the watermark passes the end of the window, giving one complete result. The Apache Beam SDK also offers processing-time triggers (emit early on a wall-clock interval) and data-driven triggers (emit after a count of elements), which trade completeness for lower latency by producing partial, possibly revised results before the window closes.

Late data is an element whose event time falls inside a window but that arrives after the watermark has passed that window's end; by default it is dropped. To keep stragglers, configure allowed lateness so the window stays open for a grace period and recomputes when a late element arrives, and pair it with an accumulating trigger so the window re-emits an updated result. Widening the window changes the aggregation boundaries instead and does not address out-of-order arrival.

Trap Enlarging the window to capture out-of-order events; that changes the aggregation grouping, whereas allowed lateness is what keeps stragglers in their correct window.

Cleansing lives inside the transform, and the durable pattern for code pipelines is the dead-letter route: a Dataflow pipeline sends records that fail parsing or validation to a side destination (a separate table or topic) instead of throwing, so one malformed row does not crash the whole job and the bad records are preserved for inspection. The principle is fail the row, not the run, and it applies whether you cleanse with Beam transforms, Wrangler, or SQL.

Trap Letting a parse failure throw and abort the batch; a dead-letter sink isolates the bad record while the rest of the data keeps flowing.

Datastream is a serverless change data capture (CDC) and replication service that streams inserts, updates, and deletes from an operational database to a destination with minimal latency, rather than re-loading whole tables. Sources include Oracle, MySQL, SQL Server, and PostgreSQL; destinations are BigQuery and Cloud Storage. Its signature use is keeping a BigQuery analytics copy current with a production database, distinct from Database Migration Service, which lands a managed Cloud SQL or AlloyDB database you then operate.

Trap Choosing Database Migration Service to feed analytics; DMS migrates the database to operate it, while Datastream's CDC feeds an in-sync analytics copy.

BigQuery Data Transfer Service (DTS) automates scheduled, managed, no-code movement of data into BigQuery, where the destination is always BigQuery. Sources span warehouses and databases (Redshift, Teradata, Snowflake, Oracle), object stores (Cloud Storage, Amazon S3, Azure Blob), and SaaS feeds (Google Ads, GA4, YouTube, Salesforce). A transfer can be a one-time backfill or a recurring scheduled load, so reach for DTS when the requirement is repeatable ingestion into BigQuery without writing pipeline code.

Storage Transfer Service moves object data online into Cloud Storage from Amazon S3, Azure Blob Storage, HDFS, on-premises file systems via agents, and other Cloud Storage buckets, with automatic retries and scheduled or one-time runs. It is optimized for transfers larger than 1 TiB; for a small one-off copy use gcloud storage instead of standing up a transfer job. Pick the mover by destination: Storage Transfer lands objects in Cloud Storage, BigQuery DTS lands rows in BigQuery.

Trap Standing up Storage Transfer Service for a sub-1-TiB ad-hoc copy; gcloud storage is the lighter tool below its optimized threshold.

BigQuery ML lets you create and run machine-learning models inside BigQuery using standard SQL, so enrichment adds a predicted column without data leaving the warehouse. You train with CREATE MODEL and score with the ML.PREDICT function over a query, which returns the input rows plus the prediction. It fits a batch, SQL-driven transform when the data already lives in BigQuery and the team is SQL-fluent rather than ML engineers.

When enrichment must happen as events flow rather than in a scheduled batch, call a deployed Vertex AI endpoint from inside a Dataflow transform: each element is sent for a prediction and the scored element continues down the pipeline. This is the streaming counterpart to BigQuery ML's ML.PREDICT, and it is the answer when the scenario scores data in real time, such as fraud checks on a transaction stream, instead of over a static BigQuery table.

Trap Reaching for BigQuery ML to score a live event stream; ML.PREDICT runs over data already in BigQuery, so real-time scoring calls a Vertex AI endpoint from Dataflow.

Beyond models trained in BigQuery, BigQuery ML can invoke a registered remote model, including a Vertex AI model or a Gemini large language model, through ML.PREDICT or ML.GENERATE_TEXT. This keeps LLM enrichment such as summarizing or classifying text inside the SQL pipeline, so you add a generated column without exporting the data. It is the in-warehouse path to generative enrichment for SQL-fluent teams.

The Eventarc trigger's location must match the location (region or multi-region) of the Cloud Storage bucket it watches. The destination Cloud Function/Cloud Run service can live in a different region, but a location mismatch on the trigger itself causes deployment errors; deploying the function in the same region too minimizes latency.

Trap Creating the trigger in the function's region instead of the bucket's region, which fails with a location-mismatch error.

Cloud Storage triggers deliver events through Pub/Sub, so the Cloud Storage service agent must hold roles/pubsub.publisher on the project, and the trigger's service account needs roles/run.invoker (and roles/eventarc.eventReceiver) to invoke the target. Having only Cloud Functions Developer or Eventarc Event Receiver is not enough and yields permission-denied at deploy time.

Trap Granting only the Eventarc Event Receiver role and omitting Pub/Sub Publisher on the Cloud Storage service agent.

google.cloud.storage.object.v1.finalized fires when an object is created or overwritten (a new generation). archived fires when a live version becomes noncurrent due to overwrite or deletion in a versioning-enabled bucket. deleted fires when an object is permanently removed. Match the event type to the exact change you need to react to.

Trap Using the deleted event to catch overwrites in a versioned bucket, when archived is the event for a version becoming noncurrent.

Cloud Functions triggered by events get at-least-once delivery, so the same event can arrive more than once, especially with retries enabled. Prevent duplicate writes by using the CloudEvent ID as an idempotency key: record or transactionally check it (e.g. in Firestore) before processing, or check whether the output already exists. Enable retries for reliability, and add an age check to stop endless retry loops on poison events.

Trap Enabling retries for reliability without an idempotency key, which turns transient failures into duplicate records.

A Cloud Storage bucket allows only 10 notification configurations per event type, so the 11th direct-trigger function fails. The fix is one notification configuration that publishes to a single Pub/Sub topic, then have every consumer function subscribe to that topic, which supports far more subscribers and lets functions filter (e.g. by file extension) in code.

Trap Adding yet another per-function Cloud Storage notification, which hits the same 10-per-event-type ceiling.

When one side of a Beam join is much smaller than the other and fits in worker memory (e.g. an MB-scale catalog enriching TB-scale transactions), load the small one as a side input and look it up inside a ParDo. The side input is cached in memory and reused across ParDos with no shuffle. Use CoGroupByKey only when both datasets are too large to fit in memory.

Trap Using CoGroupByKey to attach a tiny reference table, forcing an expensive shuffle a side input avoids.

When a Cloud Data Fusion pipeline is slow because heavy JOINs run as Apache Spark on Dataproc, enable Transformation Pushdown so supported transformations (notably complex joins) execute in BigQuery instead. This improves performance for join-heavy pipelines without rewriting the pipeline logic.

Trap Scaling up the Dataproc cluster to speed slow Data Fusion joins, instead of pushing the joins down to BigQuery.

Cloud Data Fusion ships a default plugin set; connectors that are absent from the Studio palette (Teradata, Salesforce, SAP, etc.) are added by deploying them from the Hub into your namespace, after which they appear in the palette in that namespace. No custom plugin development is needed for these pre-built connectors.

Trap Assuming a connector missing from the palette is unsupported, instead of deploying it from the Data Fusion Hub.

In a Cloud Data Fusion pipeline, source plugins read from databases and files, analytics plugins perform joins and aggregations (the Joiner plugin combines sources on a key, Group By performs sum/count/avg per group), and sink plugins write the output (e.g. to BigQuery). Joining two sources is an Analytics-category operation, not a source or sink one.

Trap Looking for a join under source or sink plugins, when joining lives in the Analytics category (Joiner).

The storage choice follows the workload's access pattern, not the data's surface shape. Classify it on three axes first: transactional (OLTP, many small consistent reads and writes) versus analytical (OLAP, large scans that aggregate); the data model (relational, key-value or wide-column, document, in-memory, or blob); and the non-functional needs (throughput, latency, total scale, regional versus global). Only then does one managed service fall out. When two services seem to fit, a non-functional need is almost always the tie-breaker.

For an OLTP relational workload (MySQL, PostgreSQL, or SQL Server) that fits in one region, Cloud SQL is the default managed choice; it scales vertically and with read replicas inside a region. Reach past it only when a hard non-functional need appears: global write scale points to Spanner, and much higher single-region PostgreSQL performance points to AlloyDB.

Trap Reaching for Spanner for a small single-region app; its global strong consistency carries a higher cost floor than a single Cloud SQL instance and is wasted when there is no global-scale need.

AlloyDB for PostgreSQL is a fully managed, PostgreSQL-compatible database with a built-in columnar engine that accelerates analytical queries by holding data in memory in columnar format. Choose it when a single-region PostgreSQL app needs significantly higher transactional and analytical performance than Cloud SQL but does not need cross-region writes, so you gain throughput without taking on Spanner's global cost floor.

Trap Jumping to Spanner for a single-region performance problem; AlloyDB raises PostgreSQL performance without the multi-region price, and Spanner is only warranted when global scale or a 99.999% SLA is genuinely required.

Firestore is a serverless NoSQL document database with automatic scaling, real-time synchronization, and offline support, built for mobile, web, and server apps. Pick it when clients must see live-updating hierarchical documents and keep working offline. It is not a warehouse, so analytical scans over its data belong in BigQuery, not Firestore.

Trap Choosing Cloud SQL for a mobile app that needs live client sync and offline mode; a relational database has no built-in real-time document synchronization, which is Firestore's signature.

Memorystore is the managed in-memory store for Redis and Memcached, used as a caching and session layer for low-latency reads of hot data such as sessions and frequently-read keys. It sits in front of a durable store to cut database load and latency; because in-memory data is not durable, the authoritative copy always stays in a database or Cloud Storage.

Trap Treating Memorystore as durable primary storage; an in-memory cache can lose data, so using it as the system of record risks losing the authoritative copy.

Analytical questions (dashboards, ad-hoc SQL over billions of rows, in-warehouse ML) select BigQuery, the serverless columnar warehouse, because it is built to scan and aggregate at scale. Running that analytics on Cloud SQL or Spanner scans too slowly and competes with live transactions; the correct pattern keeps the operational store for OLTP and replicates to BigQuery for OLAP, often via Datastream.

Trap Answering an analytics-over-huge-tables stem with the transactional database it already lives in; OLTP stores are tuned for small consistent operations, not full-table aggregation.

Unstructured and semi-structured blobs (images, logs, backups, Parquet/Avro files, a data-lake landing zone) belong in Cloud Storage, which stores objects at any scale with no up-front schema. Cloud Storage itself does not run SQL: to query those files in place you put a table interface over them with BigLake or a BigQuery external table, rather than copying the data into a database.

Trap Expecting to run SQL against Cloud Storage objects directly; object storage serves get and put, and querying files in place needs a BigLake or external-table layer over them.

Cloud Storage has four classes (Standard, Nearline, Coldline, Archive) with identical low-latency access and eleven-nines (99.999999999%) annual durability; they differ only in cost profile and minimum storage duration: Standard none, Nearline 30 days, Coldline 90 days, Archive 365 days. Colder classes cost less to store but more to retrieve, and deleting or rewriting an object before its minimum duration still incurs the full minimum charge, so match the class to how often the data is actually read.

Trap Assuming colder classes are slower to read or less durable; access latency and durability are the same across classes, so the only real penalty is higher retrieval cost and the minimum-duration charge.

Object Lifecycle Management transitions Cloud Storage objects to colder classes (SetStorageClass) or deletes them (Delete) automatically based on conditions such as object age. A rule like move to Nearline after 30 days, Coldline after 90, delete after 365 tracks declining access frequency with no manual work, which is how you keep storage cost in line with the access pattern over time.

Trap Scripting periodic deletes and re-uploads to control cost; lifecycle rules apply class transitions and deletions automatically by age, and a delete-then-reupload can even re-trigger minimum-duration charges.

BigQuery prices storage per GB and compute separately, and a table or partition not modified for 90 consecutive days automatically drops to long-term storage at about half (roughly 50% less) the active-storage rate, with no change to performance, durability, or availability. Compute is either on-demand (per TB scanned) or capacity slots; you cut scan cost by partitioning and clustering so queries read fewer bytes.

Trap Sizing BigQuery cost by node count the way you would Bigtable; BigQuery is serverless, so its cost is bytes scanned plus stored, not provisioned nodes.

BigQuery table expiration and partition expiration delete data on a schedule, so old rows are removed automatically instead of accumulating storage cost; when a partition expires, BigQuery deletes the data in that partition. Set partition expiration on a time-partitioned table to retire stale partitions, and pair it with the automatic long-term-storage discount on data left untouched for 90 days to keep a warehouse's storage bill matched to what is still queried.

Bigtable bills on the number of nodes provisioned per cluster plus the storage used, and throughput scales roughly linearly with nodes, so you size the cluster to the required queries per second and latency. Under-provisioning nodes raises latency under load; over-provisioning wastes money. This provisioned-capacity model is the opposite of BigQuery's serverless, scan-priced billing.

Trap Treating Bigtable like a serverless store with no capacity to plan; its latency and throughput depend on the node count you provision, so an undersized cluster degrades under load.

When a single country has only two Google Cloud regions and you need both data residency and the highest availability, pick a Cloud Spanner dual-region configuration: it replicates across the two in-country regions, stays within national borders, and delivers the 99.999% availability SLA (Enterprise Plus edition). Reach for it on prompts like Germany or Japan residency for a banking or healthcare app.

Trap Choosing a Spanner multi-region configuration when the prompt mandates data stay inside one country, since multi-region spans countries and breaks residency.

In a Spanner multi-region config (for example nam3) every write is committed by the default leader region first, so cross-region writes are slow when the leader is far from the writing app. Move the default leader to the read-write region closest to where write traffic originates (and use leader-aware routing) to cut write latency; after an app failover, re-point the leader to the new active region.

Trap Adding nodes or read replicas to fix slow writes, when the latency comes from the leader region being remote from the writers, not from a capacity shortfall.

When a Spanner child table (order items, transactions) is almost always read together with its parent (orders, customers), declare it with INTERLEAVE IN PARENT and prefix the child primary key with the parent key. Spanner then physically stores child rows next to the parent row in the same split, turning the join into a local primary-key lookup and removing cross-network traffic.

Trap Leaving the related tables independent and relying on a regular JOIN, which forces Spanner to fetch rows from separate splits for the hot parent-child access pattern.

For Bigtable time-series data keyed per device/sensor/vehicle, build the row key as id#timestamp so the high-cardinality id prefix spreads writes across the key space, and append a reversed timestamp when recent-first reads matter. A timestamp-first key forces every new write to the end of the key range, hotspotting one node.

Trap Putting the timestamp at the front of the row key, which makes monotonically increasing writes pile onto a single tablet server.

Add clusters to a Bigtable instance (replication) and steer traffic with app profiles: single-cluster routing pins a workload to one cluster, isolating a batch-scan job from latency-sensitive serving without duplicating data; multi-cluster routing sends each request to the nearest available cluster and auto-fails-over, giving low latency and high availability across regions.

Trap Reaching for multi-cluster routing to isolate a noisy batch job, which still lets both workloads land on the same cluster instead of separating them.

Bigtable garbage collection is configured per column family, not per column, so split data with different retention into separate column families and give each its own age (TTL) or version policy. Combine rules with an intersection policy (delete only when ALL conditions hold, e.g. keep at least N versions AND 30 days) or a union policy (delete when ANY condition holds).

Trap Trying to set a different TTL on individual columns inside one family, since the garbage-collection policy applies to the whole column family.

Set a Cloud SQL instance's availability type to REGIONAL (Multiple zones) for high availability: it synchronously replicates writes to a standby in a second zone via regional persistent disks, so a zone outage triggers automatic failover to the same IP with zero data loss. A promoted read replica is not automatically HA, so enable HA on it separately.

Trap Treating a read replica as the zone-failure safeguard, when only REGIONAL HA provides synchronous replication and automatic failover within the region.

Survive a full regional outage by adding a cross-region read replica and promoting it (replica failover) when the primary region fails; asynchronous replication keeps RPO and RTO in minutes rather than hours, which is the threshold that warrants a cross-region failover configuration. On Enterprise Plus, designate it a DR replica and use the write-endpoint DNS name so failover redirects connections without changing connection strings.

Trap Counting on same-region HA for a regional disaster, when zonal HA does not survive the loss of the whole region and a cross-region replica is required.

For predictable BigQuery workloads, buy a (commitment-discounted) baseline slot reservation for the always-on demand and add autoscaling slots that bill only while scaled up to absorb periodic spikes. Idle baseline slots are shared automatically across reservations in the same admin project and edition when ignore_idle_slots stays false; assign a project to 'None' to send it to on-demand pricing instead.

Trap Setting ignore_idle_slots to true to let one reservation borrow another's idle slots, which actually disables idle-slot sharing rather than enabling it.

Preparing data for ML on Google Cloud falls into two lanes: structured features for predictive models, and unstructured data turned into embeddings for semantic search and RAG. The lane is set by what the model consumes, not which model you eventually train, and both share one rule, do the preparation where the data already lives rather than exporting it. Naming the lane first makes the service choice fall out: tabular work points at BigQuery ML and Feature Store, embedding work at ML.GENERATE_EMBEDDING and vector search.

BigQuery ML creates and trains a model with a CREATE MODEL SQL statement directly on a BigQuery table, so the data never leaves the warehouse and SQL-fluent analysts can build models without a separate ML stack. It removes the slowest step in most ML projects, exporting large datasets into another framework. Reach for Vertex AI custom training instead when you need a specific framework like TensorFlow or PyTorch, GPUs or TPUs, or a custom training loop.

Trap Choosing Vertex AI custom training when the data is already in BigQuery and SQL would train the model in place; you pay for an export and a second system you did not need.

BigQuery ML supports many model types from SQL, including linear and logistic regression, k-means clustering, matrix factorization, PCA, boosted trees (XGBoost), deep neural networks, and ARIMA_PLUS time-series models. That breadth is why 'analysts know SQL, build a forecast or classifier fast' resolves to BigQuery ML rather than a custom Vertex AI job. The model type is chosen in the OPTIONS(model_type=...) clause of CREATE MODEL.

When you train, BigQuery ML reserves a portion of the input as an evaluation set so it can report quality on data the model did not learn from, instead of you splitting by hand. The data_split_method and data_split_eval_fraction options control how, for example a random split or a fixed evaluation fraction, with a default that automatically holds out part of the data. Evaluating on held-out rows is what catches overfitting before the model ships.

Trap Evaluating the model on the same rows it trained on; quality looks inflated because the model has already seen those examples.

Training-serving skew is when the pipeline that builds features for training differs from the one that builds them at prediction time, so the model trains on one distribution and scores on another and accuracy drops in production. The discipline that prevents it is computing each feature once and reusing the same value for both training and serving. Even without a feature store, define each transformation once and run it for both paths rather than reimplementing it on the serving side.

Trap Reimplementing the feature logic separately in the serving path; the two implementations drift and reintroduce the skew you were avoiding.

Vertex AI Feature Store is a central repository that computes a feature once and serves the same value online at low latency and offline from BigQuery as the feature source. Because training reads the offline value and production reads the online value of the same feature, training and serving see identical features and training-serving skew is eliminated. It also lets several models reuse the same curated features instead of each team recomputing them.

Trap Treating a raw BigQuery query as an online feature store; an ad-hoc warehouse read is not built for millisecond low-latency serving the way Feature Store online serving is.

An embedding is a numeric vector that positions semantically similar items near each other in vector space, so 'cancel my plan' and 'end my subscription' land close even with no shared words. That is what lets you search unstructured data by meaning rather than keyword. Embeddings are the input to both semantic search and retrieval-augmented generation, so generating them is the first step of the whole unstructured lane.

ML.GENERATE_EMBEDDING runs a remote model that references a Vertex AI text-embedding model and returns a vector for each row in a column named ml_generate_embedding_result, called over a whole table from SQL so the text never leaves BigQuery. The newer AI.GENERATE_EMBEDDING is the same job under the AI function family. You create the remote model once with a connection, then call the function across the table to embed every row in one query.

A single embedding for a whole document blurs its distinct sections, so long text is split into smaller passages and each chunk is embedded separately. Chunking lets vector search return the precise passage that answers a question rather than a whole document, which sharpens the context fed into a RAG prompt. The chunk is the unit of retrieval, so its size trades recall of detail against how much surrounding context each result carries.

VECTOR_SEARCH finds the rows whose embeddings are closest to a query vector, doing a brute-force scan on its own. Building an index with CREATE VECTOR INDEX switches it to approximate nearest neighbour (ANN), trading a little recall for a large speed gain on big vector sets. This retrieval step is the engine of RAG: embed the question, run VECTOR_SEARCH for the closest passages, then feed those passages to a generative model.

Trap Expecting an index to speed up a tiny vector set; below the size threshold BigQuery does not populate the index and the search just runs brute force anyway.

A BigQuery vector index supports two types: IVF (inverted file), which clusters vectors with k-means, and TreeAH, built on Google's ScaNN algorithm and optimized for batch queries that process many query vectors. The distance type can be EUCLIDEAN (the default), COSINE, or DOT_PRODUCT, and it must match how the embeddings were intended to be compared. Pick TreeAH when you score many query vectors in a batch and IVF for general single-query lookups.

Trap Leaving the distance type at the EUCLIDEAN default when the embeddings are meant for cosine similarity; the search then ranks by the wrong notion of closeness.

If you create a vector index on a table smaller than 10 MB the index is not populated, and VECTOR_SEARCH silently falls back to a brute-force scan. If an indexed table later shrinks below 10 MB through deletions, the index is temporarily disabled. The threshold is on total table size, not row count, so on a trivially small corpus an index buys nothing and is expected to be skipped.

Trap Assuming a vector index always accelerates VECTOR_SEARCH; under 10 MB it is not populated, so the query runs brute force regardless.

Retrieval-augmented generation grounds a generative model in your own data: embed the user's question, use vector search to retrieve the most relevant passages, then pass those passages into the model's prompt so it answers from retrieved context instead of only its training. The retrieval quality, driven by good chunking and the right distance metric, sets the ceiling on answer quality. In BigQuery the building blocks are ML.GENERATE_EMBEDDING, CREATE VECTOR INDEX, and VECTOR_SEARCH.

An object table is a read-only table over unstructured objects in Cloud Storage that exposes each object as a row carrying its uri and metadata, so you can run BigQuery ML inference or generative AI functions over images and documents and join the results to structured tables, all from SQL. The data stays in the bucket; the table only indexes it. Like a BigLake table, an object table uses access delegation, so users query it without direct read access to the underlying bucket.

Trap Loading the binary files into BigQuery to analyze them; an object table reads them in place in Cloud Storage instead of copying them into the warehouse.

A model trained with BigQuery ML can be registered to the Vertex AI Model Registry, the versioning and governance point between build and serve, so it can later be deployed to an endpoint for online prediction. This bridges the SQL-trained model into the Vertex AI serving stack without rebuilding it. It is the path when analysts train in BigQuery ML but the model must serve real-time predictions through a managed endpoint.

Both BigQuery VECTOR_SEARCH and Vertex AI Vector Search implement approximate nearest neighbour on the same ScaNN-style technology; the split is analytical versus serving. Use VECTOR_SEARCH when the vectors live in BigQuery alongside your other data and you query them with SQL. Use Vertex AI Vector Search when you need low-latency ANN serving over very large vector sets as a standalone online endpoint, typically the retrieval tier of a production RAG application.

Trap Standing up Vertex AI Vector Search when the embeddings already sit in BigQuery and an in-warehouse VECTOR_SEARCH would serve the analytical query without a second system.

When BigQuery ML does not cover the framework or you need GPUs and a custom loop, the data-engineering job is to shape features upstream and hand them to Vertex AI rather than train in SQL. Use BigQuery for structured feature transforms and Dataflow for large-scale conversion of unstructured data into training formats, then point Vertex AI custom training at the prepared data. This keeps the heavy data prep on the warehouse and the stream engine while the framework training happens on the ML platform.

Preprocessing defined in the CREATE MODEL TRANSFORM clause is stored with the model and reapplied automatically during ML.PREDICT and ML.EVALUATE, which is what prevents training-serving skew. Analytic preprocessing functions used there require an OVER() clause so the statistics learned during training are reused at serving.

Trap Computing features in a separate query before training, leaving prediction to recompute them inconsistently.

ML.FEATURE_CROSS(STRUCT(col_a, col_b, ...)) creates feature crosses—every combination of the supplied categorical columns—to capture interaction effects one feature alone can't express. Placed in the TRANSFORM clause, the cross is generated automatically during prediction as well as training.

Trap Passing the columns as bare arguments instead of wrapping them in a STRUCT as ML.FEATURE_CROSS requires.

ML.QUANTILE_BUCKETIZE splits a continuous column into buckets based on quantiles computed from the training data, so each bucket holds roughly equal counts—well suited to skewed columns with outliers. Used in the TRANSFORM clause with OVER(), the same quantile boundaries are reapplied at prediction.

Trap Reaching for fixed-width bucket boundaries when the distribution is skewed and equal-population quantile buckets are wanted.

Setting DATA_SPLIT_METHOD='CUSTOM' with DATA_SPLIT_COL lets BigQuery ML split on a column you already populated. With a BOOL column, TRUE or NULL rows go to evaluation and FALSE rows to training; with hyperparameter tuning a STRING column accepts 'TRAIN', 'EVAL', and 'TEST'. The split column is excluded from the model's features.

Trap Assuming TRUE rows in a custom BOOL split feed training—TRUE (and NULL) actually mark evaluation rows.

The Apache Beam DataFrames API offers a pandas-like interface so data scientists keep familiar syntax while Beam distributes the work across workers; running the pipeline on Dataflow provides fully managed, autoscaling infrastructure. This lets logic prototyped on a small pandas DataFrame run over datasets far too large for one machine with minimal code change.

Trap Running plain pandas, which loads the whole dataset into a single machine's memory and won't scale to terabytes.

MLTransform bundles multiple Apache Beam ML preprocessing operations (ComputeAndApplyVocabulary for categorical-to-index, ScaleToZScore for z-score normalization, TFIDF, etc.) in one class. Writing to write_artifact_location during training saves the learned vocab and normalization stats, and read_artifact_location reapplies the same transforms at inference, eliminating training-serving skew.

Trap Recomputing vocabulary and scaling stats at serving time instead of reading the artifacts saved during training.

Observability gives three signal kinds and each answers a different question: metrics (numeric time series in Cloud Monitoring) tell you whether a rate or backlog is wrong now, logs (text events in Cloud Logging) tell you what failed and why, and per-service job introspection (BigQuery INFORMATION_SCHEMA.JOBS, the Dataflow job-metrics tab, the Airflow UI) tells you which run or stage is at fault. Reach for the source that fits the question instead of scrolling one console; a growing backlog is a metric, its stack trace is a log, and the straggler stage is job introspection.

Cloud Monitoring alerting policies fire on metric thresholds, not on log text, so to page on a specific log event you create a log-based metric over a log filter and then attach an alerting policy to that metric. A counter log-based metric counts entries matching the filter; a distribution log-based metric extracts a number from each matching entry (such as a parsed latency) and builds a histogram. The flow is always log filter, then metric, then alert.

Trap Trying to attach an alerting policy directly to a log entry; Cloud Monitoring conditions evaluate metrics, so the log must become a log-based metric first.

Cloud Logging keeps entries for as long as the log bucket they land in is configured to, not per entry. The _Default bucket retains everything for 30 days, the _Required bucket holds Admin Activity and required audit logs for a fixed 400 days, and a user-defined bucket can be set from 1 day up to 3650 days (10 years). To keep audit logs past 400 days you route them through a Log Router sink to a custom bucket or to Cloud Storage.

Trap Assuming the _Required audit-log bucket's 400-day retention is configurable; it is fixed, so longer audit retention needs a sink to another bucket or Cloud Storage.

Every Cloud Logging entry passes through the Log Router, which evaluates sinks; each sink has an inclusion filter and one destination. A sink can write to a Cloud Logging bucket (keep and search), a BigQuery dataset (SQL analysis), a Cloud Storage bucket (cheap long-term archive), or a Pub/Sub topic (stream to a third-party SIEM or custom processor). Choosing the destination by intent is how you both retain and analyze the right logs.

Trap Picking a Cloud Storage sink when the goal is to run SQL over the logs; Cloud Storage is cheap archive, but querying needs the BigQuery dataset destination.

Google SRE's four golden signals to monitor for any user-facing service are latency, traffic, errors, and saturation. Anchor alerts to these symptoms and to an SLO rather than to an internal cause such as a single CPU spike, because a CPU blip that no user feels is noise while a latency or error climb is the thing worth paging on.

Error Reporting reads log entries from running services, recognizes stack traces, and groups errors by exception type and the top stack frames, so a single recurring failure surfaces as one group rather than thousands of identical lines, and it can notify on a new error. It samples up to about 1,000 errors per hour and estimates counts beyond that, so treat the count as an estimate at high volume.

Query INFORMATION_SCHEMA.JOBS (and its JOBS_BY_USER, JOBS_BY_FOLDER, JOBS_BY_ORGANIZATION views) with SQL to see running jobs plus job history for the past 180 days. The columns that drive cost and performance triage are total_bytes_billed (the on-demand cost driver), total_slot_ms (slot-time consumed), job_type/statement_type, state, and error_result for why a job failed. A scheduled query summing total_bytes_billed by user_email is the standard who-is-spending report.

Trap Reaching for Cloud Billing Reports to find which BigQuery query was expensive; Billing Reports attributes by service and SKU, but the per-query culprit lives in INFORMATION_SCHEMA.JOBS.

The query plan and execution details (Query Insights) break a query into stages reporting slot wait time, shuffle bytes, and records read versus written. High slot wait time means slot contention, a concurrency problem fixed by adding slots; large shuffle bytes mean a big join or data skew, fixed by denormalizing; records read far larger than the result means scanning too much, fixed by partitioning, clustering, or a tighter filter. The symptom in the plan picks the remedy.

Slots are processing capacity, so they speed up running a query but never change how many bytes a poorly-laid-out query must read. A single slow query is therefore usually a layout problem (no partition or clustering, a wide scan) rather than a slot shortage, and the fix is to reduce bytes scanned, not to buy capacity. Slots address concurrency and many-query contention, not one query's data volume.

Trap Buying more slots to fix one slow query whose plan shows a huge scan; capacity does not reduce bytes read, so the scan stays just as large.

In the Dataflow job-metrics tab, data freshness is the gap between an element's event-time timestamp and when it is processed, and system latency is the current maximum seconds an element has been processing or waiting. Rising data freshness together with rising backlog seconds (the estimated time to drain the current backlog) is the classic can't-keep-up picture; check the autoscaling chart to see whether workers are scaling toward the target or have hit their ceiling.

Trap Reading high data freshness as stale source data; it measures how far behind processing is, so a rising value means the job cannot keep up, not that the input aged.

A slow or stuck subscriber shows up as a growing Pub/Sub subscription backlog, and the subscription/oldest_unacked_message_age metric reports how stale the oldest unacknowledged message is. Watching it tells you the source side is falling behind before the downstream pipeline visibly stalls, which complements the Dataflow data-freshness view of the processing side.

A quota is a hard limit: when a request would exceed it the system blocks the request and the operation fails, so quotas can actually prevent consumption. A Cloud Billing budget is the opposite, notification-only, and does not cap or stop spending; its default thresholds alert at 50%, 90%, and 100% of the amount you set against actual or forecasted cost. If you must stop runaway usage the lever is a quota; a budget just tells you it is happening.

Trap Setting a Cloud Billing budget to cap or halt spending; a budget only sends alerts and never stops usage, so spend continues past the threshold.

A quota-exceeded error means a hard limit was reached, so retrying the same request keeps failing; the fix is to request a quota adjustment in Cloud Quotas, in most cases at the project level, and adjustment requests are subject to review with an email acknowledging receipt. This differs from a rate limit (a per-second or per-minute cap that smooths bursts), where client-side retry with exponential backoff is the correct response.

Trap Answering a hit absolute quota with retry-and-backoff; backoff handles transient rate limits, but an exhausted hard quota only clears with an increase request.

Because a budget cannot stop spend on its own, route its alerts to a Pub/Sub topic so a subscriber (for example a Cloud Function) can take an automated response such as disabling billing on a project. The budget still only notifies; the Pub/Sub-triggered code is what actually acts, which is the supported way to approximate a hard spending stop.

When a budget warns that cost rose, Cloud Billing Reports is the built-in visualization that breaks spend down by project, service, SKU, location, label, and time, with forecasted as well as actual figures. It answers which service or SKU drove the spend, while a per-query cost answer for BigQuery comes from INFORMATION_SCHEMA.JOBS, not from Billing Reports.

When BigQuery runs on capacity (editions) pricing, queries draw slots from reservations, and the administrative resource charts show slot usage, job concurrency, and job execution over time. Use them to confirm whether queries are waiting on slots at the project level, which is the same slot-wait signal the per-query plan reports, viewed across the whole reservation.

The immediate operational action for a job consuming far more than expected is to cancel that specific BigQuery or Dataflow job. To prevent recurrence you route heavy batch and interactive queries into separate reservations so one workload does not starve the other, but sizing those reservations is capacity planning; here the in-flight fix is the cancel and the routing is the observe-and-act follow-up.

A Cloud Monitoring metrics scope lets one scoping project view metrics from many Google Cloud projects, and even AWS accounts, so you watch an entire data platform from a single dashboard rather than hopping per project. Pair it with alerting policies and notification channels (email, PagerDuty, Slack, Pub/Sub, the mobile app) to centralize both viewing and paging.

The additionalUserLabels pipeline option attaches labels (department, environment, business unit) to each Dataflow job; the labels flow into the Cloud Billing export to BigQuery, where you filter and aggregate cost by label. This needs no extra billing configuration and the labels also filter jobs in the Dataflow monitoring UI.

Trap Trying to attribute spend per job from the billing console alone, without first stamping jobs with additionalUserLabels.

JOBS_TIMELINE has one row per second of every job's execution, so it is the most granular view of slot usage: period_slot_ms shows slot-milliseconds consumed per job per second, and period_estimated_runnable_units greater than zero means the job wanted more slots than were available. The plain JOBS view only summarizes a job, not its second-by-second timeline.

Trap Querying INFORMATION_SCHEMA.JOBS for slot contention, which lacks the per-second timeline that JOBS_TIMELINE provides.

The native bigquery.googleapis.com/query/execution_times metric feeds a Cloud Monitoring alerting policy directly. Aggregate it at the 99th percentile so an alert fires when consistently slow queries cross a threshold while ignoring occasional outliers.

Trap Building a log-based metric from query logs to track latency, when execution_times is already an alertable native metric.

Default Dataproc cluster and job metrics are collected free. Custom open-source metrics (yarn, spark, hdfs, and more) require the --metric-sources flag at cluster creation and are billed, so use --metric-overrides to collect only the critical ones for important clusters. These custom metrics carry a custom.googleapis.com/ prefix and appear in Metrics Explorer under the VM Instance resource, not under Cloud Dataproc Cluster.

Trap Hunting for custom OSS metrics under the Cloud Dataproc Cluster resource, where only the standard dataproc.googleapis.com metrics live.

Jobs submitted through the Dataproc Jobs API write driver and YARN container logs to Cloud Logging under the Cloud Dataproc Job resource type; filter by job_id and select yarn-userlogs to read Spark executor output. Cluster-level events (agent, startup) live under the Cloud Dataproc Cluster resource instead.

Trap Looking for Spark executor logs under the Cloud Dataproc Cluster resource, where only cluster-level entries appear.

A Cloud Logging sink writes through a unique writer-identity service account that has no permissions by default. After creating the sink you must grant that identity the destination's write role: Pub/Sub Publisher on a topic, or Storage Object Creator on a bucket (least privilege for compliance). Without it the sink errors and entries never arrive.

Trap Granting broad project roles or your own account access, instead of giving the sink's own writer identity the destination write role.

The Google-recommended Airflow alerting pattern is to write structured messages from on_failure_callback (DAG-level for run failures, task-level via default_args for task failures) and sla_miss_callback, then create Cloud Monitoring log-based alerting policies that match those messages and route to channels like PagerDuty. The sla parameter is set on the task as a timedelta, but sla_miss_callback must be set at the DAG level, not in default_args.

Trap Putting sla_miss_callback in default_args or on the task, where Airflow never invokes it; only the DAG-level callback fires.

When many short tasks finish on existing workers before freshly autoscaled workers warm up, the new workers sit idle and waste money. Raise the minimum worker count and tune [celery]worker_concurrency so existing workers absorb the queue instead of triggering new workers; a persistently high queued-task count with the environment pinned at its worker maximum instead means too little capacity, so raise the maximum workers.

Trap Lowering the maximum worker count to curb idle workers, which starves real bursts of capacity rather than fixing warm-up lag.

Tasks marked Failed with no logs, alongside 'Warm shutdown' and 'Received SIGTERM' in worker logs, indicate the worker pod was evicted for exceeding memory (logs were buffered and lost). Give workers more memory, or reduce [celery]worker_concurrency so each task has more memory and raise maximum workers to keep throughput.

Trap Treating logless task failures as code bugs, when SIGTERM and warm-shutdown messages point to a memory-evicted worker.

The Autoscaling rationale chart explains why the autoscaler scaled up, down, or held steady; for Streaming Engine, downscaling targets 75% CPU and is disabled when that can't be met. If workers are below maxNumWorkers with a growing backlog, the usual cause is hitting Compute Engine CPU or external-IP quota; if already at the maximum, raise maxNumWorkers with an in-flight update.

Trap Assuming a stalled scale-up is a code bottleneck, when the rationale chart and project quotas usually reveal the real cause.

When streaming data freshness climbs but backlog bytes stay normal, work items are stuck inside the pipeline rather than arriving too fast. Check the Parallel processing chart and processing_parallelism_keys for insufficient key parallelism (especially when CPU is low and even across workers), and scan worker logs for elements failing and retrying repeatedly, which holds the watermark back without a bottleneck alert.

Trap Scaling up workers when freshness rises with normal backlog, when the cause is stuck keys or retry loops, not throughput.