DEA-C01 Cheat Sheet

Sort every ingestion question by how fresh the data must be before comparing services: second-level freshness means streaming (Kinesis Data Streams or Amazon MSK), while a latency budget of minutes or hours means batch (a scheduled read from S3 with Glue or EMR). Streaming buys continuous, ordered, low-latency delivery; batch trades freshness for cheaper throughput on a bounded data set. Choosing the wrong lane, like a streaming pipeline for a nightly load, is the most common ingestion mistake.

Trap Reaching for a streaming service to satisfy a once-a-night load; batch from S3 is simpler and cheaper when second-level freshness is not required.

Kinesis Data Streams is a raw shard log that many independent consumers can read at their own offsets and re-read within the retention window, so it fits fan-out to several processors and reprocessing after a bug. Amazon Data Firehose only delivers and keeps no re-readable buffer, and MSK is the answer instead only when a team specifically needs Kafka APIs. Pick Data Streams whenever the design must replay records or feed more than one downstream application from the same stream.

Trap Choosing Amazon Data Firehose when the requirement is replay or multiple independent consumers; Firehose is a one-way delivery pipe with no re-readable retention.

Amazon Data Firehose (formerly Kinesis Data Firehose) is a fully managed, serverless delivery stream with no shards to size and no consumers to run; you point producers at it and it loads to S3, Amazon Redshift, Amazon OpenSearch Service, Splunk, or a generic HTTP endpoint. It can optionally invoke a Lambda function to transform records and convert their format in flight. Reach for it when the whole job is buffer-and-deliver and you do not need to re-read the data.

Amazon Data Firehose buffers incoming records by a buffer size in MB and a buffer interval in seconds, then delivers a batch when either threshold is reached. A larger interval produces fewer, bigger objects in S3 and adds delivery latency, while a smaller interval delivers sooner in smaller files; tune the two to trade freshness against object count. This buffering is why Firehose is near-real-time rather than truly real-time.

Each Kinesis Data Streams shard ingests up to 1 MB/s or 1,000 records per second on writes, whichever it reaches first, while a single record's data payload can be up to 10 MiB (the shard sustains 1 MB/s and absorbs the occasional large record with burst capacity). Exceed the write ceiling and the API returns ProvisionedThroughputExceededException, so size shard count to peak throughput in provisioned mode. Producers should aggregate or batch records and retry with exponential backoff to stay under the per-shard cap.

A shard's classic (shared) read path serves 2 MB/s total split across all polling consumers, capped at five GetRecords calls per second, so adding consumers starves them. Enhanced fan-out (EFO) gives each registered consumer a dedicated 2 MB/s per shard pushed over HTTP/2 via SubscribeToShard, which also lowers latency. Use EFO when several applications must each read the full stream at full speed; stay on shared reads when one or two consumers fit inside 2 MB/s, since EFO bills per consumer-shard hour plus data retrieved.

Trap Assuming adding a third or fourth classic consumer scales read throughput; they all divide the same shared 2 MB/s and throttle, which is exactly what EFO's dedicated 2 MB/s per consumer solves.

You can register up to 20 enhanced fan-out consumers for each data stream on On-Demand Standard and Provisioned modes (On-demand Advantage mode allows 50). The limit is per stream, not per shard, and each registered EFO consumer still gets its own dedicated 2 MB/s on every shard. Confusing the unit is a classic distractor.

Trap Reading the EFO limit as 20 consumers per shard; it is 20 registered consumers per stream (50 in On-demand Advantage mode), independent of shard count.

Kinesis on-demand capacity mode removes shard sizing entirely: a new on-demand stream starts at 4 MB/s write and 8 MB/s read and scales automatically with traffic. It is the right call for spiky or unpredictable workloads where you would otherwise over- or under-provision shards. Provisioned mode is cheaper at steady, well-understood throughput where you can right-size shard count. You can switch a stream between on-demand and provisioned modes twice within 24 hours.

A Kinesis data stream keeps records for a configurable retention period, defaulting to 24 hours and extendable to a maximum of 8,760 hours (365 days). Within that window a consumer can reset its iterator to an earlier sequence number and re-read, which is the mechanism behind replayability after a downstream bug or outage. A Firehose-only pipeline has no such window, so to combine managed delivery with replay you put Kinesis Data Streams in front of Firehose.

AWS Database Migration Service (DMS) connects to relational, NoSQL, and warehouse sources and does both a one-time full load and ongoing change data capture (CDC) that replicates inserts, updates, and deletes to keep the target in sync. It supports fully heterogeneous migrations, so pair it with the AWS Schema Conversion Tool (SCT) when source and target engines differ, for example Oracle to Aurora PostgreSQL. DMS targets include S3, Kinesis, Amazon Redshift, and DynamoDB, so it is also how you stream a database's change feed into a data lake.

Trap Hand-rolling a polling job against the source database to capture changes; DMS CDC reads the database's transaction log and replicates inserts, updates, and deletes natively.

Amazon AppFlow is the no-code, fully managed connector for SaaS APIs such as Salesforce, ServiceNow, Zendesk, and Slack, running flows on a schedule, on an event, or on demand and landing data in S3 or Amazon Redshift. When a question says to pull data from a named SaaS app, AppFlow is the answer rather than a custom API client or a generic file transfer service.

Trap Writing a custom API client (or using DataSync) to pull from Salesforce; AppFlow is the managed SaaS connector built for exactly that source.

AWS DataSync moves large volumes of files or objects between on-premises storage (NFS, SMB, HDFS, object stores) and AWS (S3, EFS, FSx) with scheduling, verification, and high throughput, so it fits recurring bulk file ingestion you control. AWS Transfer Family is the inbound counterpart: it provides managed SFTP, FTPS, or FTP endpoints so external partners can push files straight into S3 or EFS without you running a file server. The deciding question is who initiates the transfer and over which protocol.

Trap Standing up a Transfer Family SFTP server to pull files you control from on-premises NAS; that is DataSync's job, while Transfer Family exists for partners pushing files in over SFTP/FTPS/FTP.

When transferring the data set over the available bandwidth would take an unacceptable amount of time, ship it physically with AWS Snow Family (Snowball Edge) devices. Snowmobile is retired, so for multi-petabyte transfers answer with Snowball Edge fleets, AWS Direct Connect, or DataSync over a high-bandwidth link. The trigger phrase is a large volume plus limited bandwidth or a hard deadline.

Trap Answering a multi-petabyte transfer with Snowmobile; Snowmobile is retired, so the offline answer is Snowball Edge fleets (or Direct Connect / DataSync for an online link).

Event-driven ingestion fires the moment data lands: an S3 Event Notification on s3:ObjectCreated:* can target Lambda, SNS, SQS, or EventBridge, which then starts a Lambda function, a Glue job, or a Step Functions workflow. Use events when latency matters and arrivals are irregular. Use a schedule (EventBridge Scheduler, a Glue trigger, or MWAA) instead when work is periodic and batched.

To process a Kinesis (or DynamoDB) stream with Lambda, configure an event source mapping: Lambda polls the stream on your behalf and invokes the function per batch of records, with the batch size and parallelization you set. This is distinct from an S3 Event Notification, which fires on object creation in a bucket. When a stem says to call a Lambda from a Kinesis stream, the answer is the event source mapping.

Trap Wiring an S3 Event Notification to invoke Lambda from a Kinesis stream; S3 notifications fire on bucket object events, while a Kinesis-to-Lambda integration is an event source mapping that polls the shards.

Replayability is the ability to re-process records after a failure without data loss, and the exam rewards designs that have it. A Kinesis stream is replayable within its retention window by resetting the iterator, and an S3 landing zone is replayable by construction because a failed batch job just re-runs against the same immutable objects. DynamoDB Streams gives a 24-hour replay window over item-level changes for the same reason; Amazon Data Firehose alone cannot replay, so front it with Kinesis Data Streams when you need both managed delivery and replay.

A stateless transaction processes each record independently, so retries and parallel consumers are safe; a stateful transaction depends on order or accumulated context. Kinesis preserves order only within a shard, so to process stateful records in sequence you assign a partition key that keeps related records (for example, all events for one customer) on the same shard. Spreading them across shards parallelizes throughput but loses cross-shard ordering.

Trap Assuming a Kinesis stream preserves global ordering across all shards; ordering is guaranteed only within a shard, so stateful, order-sensitive records need a partition key that pins them to one shard.

Downstream targets throttle on their own limits: a Kinesis shard at 1 MB/s or 1,000 records/s, and DynamoDB and RDS on their provisioned capacity. Producers absorb this by catching ProvisionedThroughputExceededException and retrying with exponential backoff, by batching with PutRecords, and by aggregating small records so fewer, larger writes stay under the cap. Throttling is a signal to scale capacity (more shards or on-demand mode) or smooth the producer, not to retry tighter.

DynamoDB Streams captures a time-ordered, exactly-once log of item-level inserts, updates, and deletes and retains it for 24 hours. The StreamViewType controls what each record carries: KEYS_ONLY, NEW_IMAGE, OLD_IMAGE, or NEW_AND_OLD_IMAGES, and it cannot be changed after the stream is created. Consume it with a Lambda trigger, Apache Flink, or the Kinesis adapter to drive change-driven ingestion such as replicating updates into a data lake.

Trap Expecting to switch a DynamoDB stream's StreamViewType (for example KEYS_ONLY to NEW_AND_OLD_IMAGES) in place; the view type is fixed at creation, so you must disable the stream and create a new one.

When a full load was done separately (native tools or a prior full-load task), create a CDC-only DMS task and set CdcStartPosition to a native start point so replication resumes exactly where the load finished with no gap or overlap. For Oracle the native start point is the System Change Number (SCN); for MySQL and MariaDB it is the binary log file name and position written as file:position (e.g. mysql-bin-changelog.000024:373).

Trap Starting the CDC task from a wall-clock timestamp, which can skip or replay transactions because one timestamp can map to multiple native points in the log.

Amazon S3 Event Notifications cannot target an SQS FIFO queue; standard queues, SNS, Lambda, and EventBridge are the only direct destinations. To preserve strict ordering, enable EventBridge notifications on the bucket and add a rule whose target is the FIFO queue, since EventBridge supports sending S3 events to a FIFO queue.

Trap Configuring an S3 event notification to point straight at a .fifo queue, which the bucket configuration rejects.

When a Lambda triggered by S3 writes its output back to the same bucket, the new PutObject fires another notification and creates a recursive invocation loop. Break the loop by scoping the trigger with a prefix or suffix filter so only input objects invoke the function, or by writing transformed output to a different bucket or prefix.

Trap Triggering on the whole bucket and writing results back into it, so each transformed object launches the function again.

Plain S3 Event Notifications match only on prefix and suffix and allow one destination per event type, whereas EventBridge can filter on rich metadata such as object key suffix and numeric object size and route each pattern to its own rule with its own targets. Enable EventBridge on the bucket to send all events to the default event bus, then add multiple independent rules to route different file types or sizes to different consumers.

AWS Glue runs Apache Spark with no cluster to manage and bills per second, so it fits scheduled extract-transform-load jobs that run and then stop. It integrates with the Glue Data Catalog and offers job bookmarks for incremental loads, which is why it beats a standing EMR cluster for short, bursty jobs that would otherwise leave a cluster idle and billing. Choose EMR instead when you need a specific framework version, custom cluster configuration, or the deepest Spot discount on long batches.

Trap Standing up a long-lived EMR cluster for short nightly jobs; the cluster bills around the clock while idle, whereas Glue bills only while a job runs.

Glue measures capacity in Data Processing Units (DPUs), where one DPU is 4 vCPUs and 16 GB of memory, and a job's cost is DPUs times run time. Glue bills per second with a 1-minute minimum, needs at least 2 DPUs, and allocates 10 DPUs to an ETL job by default. The per-second-with-no-idle-cost model is the core reason Glue is cheaper than an always-on cluster for intermittent work.

Glue worker types map to DPU multiples: G.1X is 1 DPU (4 vCPU, 16 GB) and is AWS's recommended default for most transforms, joins, and queries; G.2X is 2 DPU; and G.4X (4 DPU) and G.8X (8 DPU) are for the most demanding aggregations. The separate G.025X worker is 0.25 DPU and is meant only for low-volume streaming jobs, not batch ETL. Pick a bigger worker for memory-heavy joins, more workers for parallelism.

Trap Using G.025X for a batch ETL job to save money; it is a 0.25-DPU streaming worker and starves a large batch transform of memory.

A Glue job bookmark is persisted state recording what a job already processed, so a scheduled rerun handles only new data instead of reprocessing the whole dataset. For S3 sources Glue decides what is new by the object's last-modified timestamp; for JDBC sources it uses a bookmark key column, by default the primary key when it increases or decreases sequentially with no gaps. The setting has three modes: Enable (track and process new data), Disable (the default, reprocess everything), and Pause (process new data without advancing the bookmark).

Trap Leaving bookmarks at the default Disable and expecting incremental behavior; Disable reprocesses the entire dataset every run, so duplicates land downstream.

When you intend to reprocess all source data with the same job, reset the bookmark using the ResetJobBookmark API, the console, or aws glue reset-job-bookmark. Resetting clears the processed-state so the next run reads everything again, but Glue does not clean target files, since only source files are tracked, so point the rerun at a fresh output location to avoid duplicate data.

AWS Glue Studio is a visual editor where you lay out source, transform, and target nodes on a canvas and Studio generates the underlying Apache Spark code. It lowers the bar for authoring ETL while still producing a real Spark job, so the choice between Studio and a hand-written script is about authoring convenience, not a different runtime.

Amazon EMR runs the full open-source stack, Apache Spark, Hive, and Presto/Trino, on clusters you size (EMR on EC2) or on EMR Serverless. It wins over Glue when you need a specific framework version, bootstrap actions, custom libraries, or the lowest cost on long batches via EC2 Spot. EMR Serverless removes cluster sizing by auto-scaling Spark or Hive workers and releasing them when the job finishes.

EC2 Spot Instances sell spare capacity for up to ~90% off On-Demand but can be reclaimed on short notice, so on EMR you put interruption-tolerant task nodes on Spot while keeping core nodes (which hold HDFS data) and the master node On-Demand. That way a reclaimed Spot node loses only recomputable work, never cluster state or stored data. This is the canonical EMR cost-optimization answer for long, fault-tolerant batch jobs.

Trap Putting EMR core or master nodes on Spot to cut cost; reclaiming a core node can lose HDFS data and reclaiming the master kills the whole cluster.

EMR Serverless is a deployment option that auto-determines the resources a Spark or Hive job needs, schedules workers, and releases them when the job finishes, so you avoid over- or under-provisioning. For jobs that must start in seconds, pre-initialized capacity keeps a warm pool of workers ready. It is the EMR answer when you want managed open-source frameworks but no cluster operations.

AWS Lambda reshapes a small payload in response to an event, such as an S3 object created or a stream record, with no Spark cluster. It is bounded by a 15-minute maximum runtime and 10 GB of memory, so it fits per-file or per-record transforms but not multi-gigabyte joins or aggregations, which belong on Glue or EMR.

Trap Choosing Lambda for a large batch join; the 15-minute runtime and 10 GB memory ceilings make it the wrong engine for heavy Spark-scale work.

When data already lives in Amazon Redshift, reshape it in place with SQL using stored procedures and CREATE TABLE AS SELECT, the extract-load-transform (ELT) pattern. This avoids pulling data out to Glue or EMR and back. If the data is not yet loaded into Redshift, this is not the transform engine to pick.

Trap Pulling data out of Redshift into a Glue Spark job to reshape it; in-warehouse SQL ELT avoids the round trip when the data is already loaded.

Row-based CSV and JSON force an engine to read every column even when a query needs a few, while columnar Parquet and ORC let it read only referenced columns (column pruning) and skip whole chunks using min/max statistics (predicate pushdown). Converting CSV to Parquet is the canonical DEA-C01 transform task, done with a Glue or EMR Spark job, Athena CREATE TABLE AS SELECT, or Amazon Data Firehose record-format conversion for streams.

Trap Leaving curated data as CSV or JSON to query later; row formats read every column on every scan, so Athena costs and query times stay high.

Partitioning writes data into S3 prefixes keyed by a column, for example year=2026/month=06/, so a query filtered to that range lists only the matching prefix and skips the rest (partition pruning). Choose partition keys that match how queries actually filter, commonly date, and avoid over-partitioning into millions of tiny partitions, which trades scan savings for crippling metadata and listing overhead.

Trap Partitioning on a high-cardinality column like user ID; it creates millions of tiny partitions whose listing and metadata cost outweighs the pruning benefit.

Compress Parquet/ORC output with a splittable codec, Snappy (fast, common default), GZIP (smaller, slower), or ZSTD, to cut both storage and bytes scanned. Pair this with coalescing output into reasonably sized files (a few hundred MB is a common target) rather than emitting one file per record, because thousands of tiny objects force an engine to open thousands of files and inflate task overhead.

Trap Writing one output file per input record; the resulting small-files explosion makes every downstream scan open thousands of objects and slows jobs more than the data volume warrants.

Glue and EMR Spark read relational sources, RDS, Aurora, Amazon Redshift, and external on-prem databases, over JDBC connections, while BI and client tools more often connect over ODBC. A Glue connection stores the JDBC URL, credentials (kept in AWS Secrets Manager), and the VPC networking a job needs to reach a private database, so one job can join a JDBC table against S3 data into a single integrated dataset.

Defining a workload by volume (how much), velocity (how fast it arrives), and variety (structured, semi-structured, unstructured) points to the right engine. High volume and velocity push toward distributed Spark engines (Glue or EMR), while high variety, mixing tables with JSON or free text, is what makes a flexible transform engine necessary rather than plain SQL over uniform rows.

To enrich records with a large language model (classify tickets, extract fields, summarize documents), invoke Amazon Bedrock from your Glue or EMR Spark code and write the model output as a new column. Bedrock is serverless and exposes a single inference API across foundation models, so the transform engine still does the heavy data movement while Bedrock supplies the per-record intelligence.

Glue's Flex execution class runs non-urgent jobs (pre-production, testing, one-time loads) on spare capacity at lower cost, with the trade-off of slower and less predictable start times. Reach for it when a job is not time-sensitive; keep the standard execution class for time-sensitive workloads that need fast startup and dedicated resources.

Trap Putting a time-sensitive, SLA-bound job on Flex to save money; Flex trades fast startup for spare-capacity scheduling and can delay the run unpredictably.

Amazon Data Firehose buffers records before invoking the transformation Lambda using a buffer size hint between 0.2 MB and 3 MB (default 1 MB); raising it toward 3 MB batches more records per invocation while staying under Lambda's 6 MB synchronous payload limit. A transformation invocation may run up to 5 minutes, after which Firehose times out and retries. Enable source record backup to deliver the original untransformed records to a separate S3 location concurrently with the transformed output.

Trap Confusing this Lambda buffering hint with the delivery buffering that flushes to the destination on size or interval.

maximizeResourceAllocation is an Amazon EMR-specific Spark configuration that, when true, calculates the maximum compute and memory available for an executor on an instance in the core instance group and sets the spark-defaults executor, driver, and parallelism settings accordingly. It best fits a cluster running one large Spark application at a time so the job can use the whole cluster instead of leaving executors underutilized.

Trap Enabling it alongside other distributed applications such as HBase, whose custom YARN configurations conflict with maximizeResourceAllocation and can cause Spark jobs to fail.

Use aws emr add-steps with command-runner.jar to submit a Spark application to an existing EMR cluster; pass spark-submit and your options (including --deploy-mode cluster) in Args so the driver runs on the ApplicationMaster inside the cluster rather than on the client. Running the job as a step makes it independent of any SSH session and records its status and logs under the cluster's Steps section.

Trap SSHing in to run spark-submit in client mode, which ties the driver to the terminal session and loses the step's tracking and logs.

Sort every store question by the dominant access pattern first: online transaction processing (OLTP) is many small point reads and writes at high concurrency keyed by an identifier, while online analytical processing (OLAP) is a few large queries that scan and aggregate across columns. OLTP wants RDS, Aurora, or DynamoDB; OLAP wants the Redshift columnar warehouse or query-in-place over an S3 lake. The storage layout that makes one fast (row-oriented for point lookups, columnar for scans) makes the other slow, so settle the workload type before comparing services.

Trap Running warehouse-style aggregation scans on a row-oriented OLTP engine like RDS; it reads whole rows off disk and drags every column even when the query needs one.

When the data is relational and queries need joins, multi-row transactions, and ad hoc SQL, use Amazon RDS or Amazon Aurora; Aurora is the cloud-native engine that separates compute from a shared storage volume and is MySQL- and PostgreSQL-compatible. The relational trade is flexible ad hoc querying in exchange for a fixed schema and writes bound to a single primary instance (you add read replicas for read scale, not write scale). Reach here whenever the application's value is in relationships between tables rather than raw key-lookup throughput.

An Aurora cluster volume grows automatically up to 256 TiB on current engine versions (older versions topped out at 128 TiB), so you do not pre-provision storage the way you do for RDS. Many third-party sources still quote 64 or 128 TiB; use 256 TiB for current content. This matters when a question pits Aurora against another store on a capacity ceiling.

Use DynamoDB when the workload is key-value or document, the access patterns are known in advance, and you need single-digit-millisecond reads at effectively unlimited request scale with no servers to manage. It scales horizontally by partitioning on the key, so it has no instance ceiling, but you must design the partition and sort keys (and secondary indexes) around the queries before storing data. The trade versus relational is horizontal scale and predictable latency in exchange for committing to the access patterns up front.

Trap Choosing DynamoDB when analysts need to query the data many unforeseen, ad hoc ways; a query it was not key-designed for forces an expensive full scan.

Both Amazon ElastiCache and Amazon MemoryDB serve microsecond reads from RAM, but they differ in durability: ElastiCache (Redis OSS, Valkey, or Memcached) is a cache that sits in front of a database and a node loss can drop cached data, while MemoryDB stores data durably across Availability Zones with a Multi-AZ transactional log and delivers microsecond reads with single-digit-millisecond writes. Put ElastiCache in front of RDS, Aurora, or DynamoDB to absorb hot reads; choose MemoryDB only when the fast key-value store must also be durable enough to be the system of record, which lets it replace a separate cache plus database.

Trap Treating a plain ElastiCache cache as the only durable copy of the data; it is a cache in front of a database, so a node failure can lose what was only cached.

Amazon Redshift is a columnar, massively parallel processing (MPP) warehouse: it stores data by column and spreads work across nodes, so repeated aggregations over modeled, structured data run fast once you load it and set distribution and sort keys. It is the OLAP answer when dashboards and reports re-run the same heavy queries. It is not built for thousands of single-row lookups per second, which is DynamoDB's job, so match it to scan-heavy analytics rather than point access.

Trap Reaching for Redshift to serve high-concurrency point lookups; an MPP columnar warehouse is tuned for big scans, not single-row OLTP reads.

Redshift Spectrum lets Redshift run SQL directly against files in Amazon S3 by defining an external schema (backed by the AWS Glue Data Catalog) and external tables over the S3 location, then joining those external tables to local Redshift tables. The data stays in S3 and nothing is loaded into the warehouse, so it fits large, cold fact data you keep in the lake while small dimension tables live in Redshift. When a stem says "query data in S3 without loading it into the warehouse," that is Spectrum.

Trap Copying the whole S3 lake into Redshift to join against it; Spectrum reads the external tables in place, avoiding the load and duplicate storage.

A Redshift federated query reads live from operational Amazon RDS and Aurora PostgreSQL or MySQL databases, so you can join the warehouse to current transactional data without first building a pipeline to copy it. It is the opposite direction from Spectrum: Spectrum reaches out to S3 files, a federated query reaches out to a running OLTP database. Pick it when the requirement is joining analytics to up-to-the-moment operational rows.

Trap Using Redshift Spectrum to reach a live transactional database; Spectrum targets S3 files, while reading a running RDS/Aurora database is a federated query.

A Redshift materialized view precomputes and stores the result of an expensive query and can refresh incrementally, so dashboards read the cached answer instead of rescanning the base tables each time. Reach for it when the same heavy aggregation runs repeatedly and the underlying data changes slowly enough that a refreshed cache is acceptable. It speeds up reads at the cost of storing and refreshing the precomputed result.

An Amazon S3 data lake stores raw and curated files in open formats cheaply and lets many engines (Athena, EMR, Redshift Spectrum) read them in place with schema-on-read, paying per query rather than per loaded table. Use it as the durable landing zone for everything, then add a warehouse only for the curated, performance-critical queries. The lake-versus-warehouse decision is a query-engine choice over what is often the same underlying data, not an either/or.

Raw Parquet or CSV files on S3 have no transactions, no row-level updates or deletes, and no safe schema change, so an open table format like Apache Iceberg adds a metadata layer over the same objects that brings ACID transactions, snapshot isolation, time travel, hidden partitioning, and schema evolution. Choose Iceberg when many engines must read and write the same lake tables consistently or you need to update and delete individual rows. It keeps the lake's cost and openness while gaining a warehouse table's correctness.

Trap Trying to update or delete individual records in plain S3 Parquet/CSV files; they have no transactions, so you need an open table format like Iceberg for row-level mutations.

Amazon S3 Tables is the managed way to run Apache Iceberg on S3: a new table-bucket type stores tables as first-class resources, delivers higher transactions per second and query throughput than self-managed Iceberg in a general-purpose bucket, and has S3 continuously perform table maintenance for you (automatic compaction of small files, snapshot management, and unreferenced-file removal). Choose it when you want Iceberg's ACID and time-travel benefits on the lake with the least operational overhead. It is Iceberg-format storage queried by Iceberg-aware engines, not a new database engine.

Trap Assuming S3 Tables is a separate database like DynamoDB; it is Iceberg tabular storage in S3 with AWS running compaction and cleanup, queried by the same Iceberg SQL engines.

When the data has a natural non-relational shape, use the store built for it rather than bending a relational table: Amazon Neptune for graph traversals like fraud rings or social graphs, Amazon Keyspaces for Cassandra-compatible wide-column data, and Amazon DocumentDB for MongoDB-compatible documents. These are shape matches that give you the Gremlin/openCypher, Cassandra, or MongoDB APIs, not raw performance upgrades over DynamoDB. Pick them when a deep multi-hop traversal would become a chain of self-joins, or when the application already speaks one of those data models.

Trap Modeling deep relationship traversals as repeated self-joins in a relational table; a graph database like Neptune traverses edges directly and is what those queries need.

For similarity search over embeddings, keep the vectors next to the query engine you already use: Amazon Aurora PostgreSQL with the pgvector extension runs nearest-neighbor SQL beside relational data, and Amazon OpenSearch Service offers vector fields for the same job. Among PostgreSQL options, only Aurora PostgreSQL (not plain RDS for PostgreSQL) is a selectable vector store for Amazon Bedrock Knowledge Bases. Choose the vector store by where the rest of the workload runs rather than standing up a separate similarity-search system.

Trap Assuming plain RDS for PostgreSQL is a selectable Bedrock Knowledge Bases vector store; only Aurora PostgreSQL qualifies among the PostgreSQL options.

The vector index type is its own decision: HNSW (hierarchical navigable small world) builds a layered proximity graph that gives high recall and fast queries but uses more memory and is slower to build, while IVF (inverted file, IVFFlat in pgvector) partitions vectors into lists and searches only the nearest few, so it builds faster and uses less memory but can trade away some recall. Choose HNSW when query latency and recall matter most; choose IVF when index build time or memory is the binding constraint. The choice is a recall-and-speed versus memory-and-build-cost tradeoff, not a correctness one.

Exam stems telegraph the right store through the access pattern, so train on the verbs: "single-digit-millisecond key-value lookups, serverless" is DynamoDB; "relational with joins and transactions" is RDS or Aurora; "fast SQL analytics over modeled data" is Redshift; "query files in S3 without loading" is Athena or Redshift Spectrum; "join the warehouse to live operational data" is a federated query; "ACID updates and time travel on the lake with least operational effort" is S3 Tables; "microsecond reads with durability, no separate cache" is MemoryDB. The most common wrong answer is the store tuned for the opposite workload, so read the pattern before reaching for the familiar service.

On-demand capacity mode auto-scales per request with no capacity planning, making it the default and recommended choice for new or spiky workloads. Provisioned mode (optionally with reserved capacity) is more cost-effective for steady, predictable, high-utilization traffic that you can reliably forecast.

Trap Keeping provisioned capacity on a steady-but-lightly-utilized table where on-demand pay-per-request would cost less.

Amazon OpenSearch Serverless auto-provisions and scales compute (in OCUs) with demand and removes cluster capacity management, fitting infrequent, intermittent, or spiky workloads. Use the Search collection type for full-text search and the Time series collection type for log analytics.

Aurora Serverless v2 (MySQL- or PostgreSQL-compatible) scales capacity in fine-grained half-ACU increments to follow demand and can scale to zero ACUs with automatic pause and resume during idle periods, so it suits unpredictable, spiky transactional workloads while supporting Multi-AZ and standard Aurora features.

Aurora Replicas share the primary's storage volume, so routing read-heavy or reporting queries to the reader endpoint scales reads with low replica lag and no schema change, while writes stay on the writer endpoint. RDS read replicas serve the same purpose for non-Aurora engines.

Concurrency scaling adds transient cluster capacity automatically when queries start queuing on a WLM queue, giving consistent performance during peaks without permanently resizing the cluster. Set the Concurrency Scaling mode to auto on the affected WLM queue and route the bursty workload there.

Define a table once in the AWS Glue Data Catalog and Athena, Amazon EMR, and Redshift Spectrum all query it, because the catalog is Apache Hive metastore compatible. It stores metadata (database, table, schema, S3 location, partitions), not the data itself, so a table is a schema plus a pointer. When several engines must read the same S3 data with one definition, the answer is a single Glue Data Catalog table rather than a separate definition per engine.

Trap Defining the schema separately in each engine; the Hive-compatible catalog exists so one table definition serves Athena, EMR, and Spectrum at once.

Point a Glue crawler at a data store and it runs a classifier to infer the schema, then creates or updates the table and its partitions in the Data Catalog automatically. It is the right tool when columns or partitions drift and you do not want to maintain DDL by hand. For a tiny, fixed schema a crawler is overhead; define the table once with CREATE TABLE or infrastructure-as-code instead.

A crawler does not guess schema on its own; it applies a classifier that recognizes a data format such as CSV, JSON, Parquet, ORC, or Avro and produces the column names and types. Glue ships built-in classifiers and lets you add custom classifiers for formats the built-ins miss. The classifier is the schema-inference step inside the crawl, distinct from the crawler that schedules and writes to the catalog.

A Glue crawler runs on demand or on a schedule you express as a cron expression, choosing frequency, days, and time. Use a schedule to keep the catalog current as new data lands without anyone triggering the crawl manually. There is no separate always-on listener for plain scheduled crawls; the schedule itself drives recrawls.

When a large data set makes full recrawls slow and expensive, configure the crawler in Amazon S3 event mode: it consumes S3 event notifications from an SQS queue and lists only the folders that changed between crawls instead of the entire target, cutting recrawl time and cost. The first crawl is always a full listing; only subsequent crawls run incrementally from events. Reach for this when the stem stresses huge, incrementally-updated data and recrawl cost.

Trap Keeping a plain scheduled crawler that re-lists the whole bucket every run when the data set is huge; that is the costly option S3 event mode exists to replace.

To catalog a database rather than S3, create an AWS Glue connection: a Data Catalog object that stores connection information (JDBC URI, credentials, VPC details) for a data store. Crawlers and ETL jobs both reach the source or target through it, so you configure the connection once and reuse it. Use a JDBC connection to crawl Amazon RDS, Amazon Redshift, or an on-premises database into the catalog.

For highly partitioned tables, Athena normally makes a GetPartitions call to the Data Catalog before pruning, and that call is slow when partitions number in the hundreds of thousands. Partition projection removes it: Athena calculates partition values and locations from table properties you configure, replacing a remote metadata lookup with an in-memory computation and dropping query runtime. It also automates partition management, because new date or integer partitions need no registration at all.

Trap Reaching for projection to cut query cost on a table with only a handful of partitions; the GetPartitions call it removes is only a bottleneck on highly partitioned tables.

Partition projection works when partitions follow a predictable pattern: continuous integers, continuous dates or datetimes, a finite set of enumerated values (like Region or airport codes), or AWS service log layouts. You declare the value ranges and projection type per partition column in table properties. Irregular or unpredictable partition naming is not projectable, and a query for a value outside the configured range returns zero rows rather than an error.

Once partition projection is enabled on a table, Athena ignores any partition metadata registered for that table in the Glue Data Catalog or Hive metastore and projects values instead. So you do not mix projection with crawler- or MSCK-registered partitions on the same table; the projected configuration is the single source of truth. SHOW PARTITIONS also will not list projected partitions, because projection is a DML-only feature.

Trap Running a crawler or MSCK to add partitions to a table that already uses projection; Athena ignores those catalog entries, so the work has no effect.

Partition projection is usable only when the table is queried through Athena. If the same table is read by Redshift Spectrum, Athena for Spark, or Amazon EMR, those engines fall back to standard catalog partition metadata, not projection. So when a table must serve engines beyond Athena, keep real catalog partitions (and consider a partition index) instead of relying on projection.

Trap Choosing partition projection for a table that Redshift Spectrum or EMR also reads; only Athena projects, so the other engines see no partitions unless they are in the catalog.

MSCK REPAIR TABLE scans the S3 file system for Hive-compatible (key=value) partition folders added after the table was created and adds the missing ones to the catalog. It only adds partitions and never removes them, so deleting folders in S3 and re-running it will not drop stale partitions; use ALTER TABLE DROP PARTITION for that. It also handles only Hive-style layouts; non-Hive layouts need ALTER TABLE ADD PARTITION.

Trap Expecting MSCK REPAIR TABLE to remove partitions whose S3 folders were deleted; it only adds Hive-style partitions, so stale entries linger until you DROP PARTITION.

MSCK REPAIR TABLE is best for first-time setup or a bulk reconciliation of many new Hive partitions at once, because it scans all subfolders and can time out on very large tables. For frequent incremental additions (such as a new daily partition) ALTER TABLE ADD PARTITION adds the partition directly without the scan-everything cost, avoiding the query timeouts MSCK hits when run repeatedly. Match the tool to add-once-in-bulk versus add-often-one-at-a-time.

Trap Running MSCK REPAIR TABLE on every new daily partition; it rescans all subfolders and times out, where ALTER TABLE ADD PARTITION adds just the one.

When you keep catalog partitions, a Glue partition index lets GetPartitions fetch a subset of partitions rather than loading every partition and filtering in memory, which speeds queries as partition counts grow. Unlike projection, an index benefits Redshift Spectrum, EMR, and Glue ETL too, because it accelerates the shared catalog rather than skipping it. Reach for an index when many engines read a heavily-partitioned table and projection is off the table.

Trap Assuming a partition index is Athena-only like projection; the index speeds the shared catalog, so Spectrum, EMR, and Glue ETL benefit too.

A table supports a maximum of 3 partition indexes, and an index only helps when the first key of the index appears in the filter expression. The indexing solution supports the AND operator; sub-expressions using OR, IN, LIKE, or NOT are ignored by the index and filtered afterward on the narrowed set. So design indexes around your real query patterns and lead the filter with the index's first key.

Two AWS services are both called catalogs and answer different questions. The Glue Data Catalog is the technical catalog: the metastore engines query for schema, location, and partitions. Amazon SageMaker Catalog (formerly Amazon DataZone) is the business catalog: it lets people catalog, discover, share, and govern data with business glossary terms across an organization. The technical catalog is for query engines, the business catalog is for people who need to find and request data.

Trap Pointing org-wide data discovery with a business glossary at the Glue Data Catalog; that is SageMaker Catalog / DataZone, which is the business catalog.

The business catalog builds on the technical one: in DataZone you publish data assets to the catalog from data already in the AWS Glue Data Catalog and from Amazon Redshift tables and views, and you can manually publish S3 objects. Analysts then search by business terms, request access through a governed subscribe workflow, and analyze with Athena or Redshift query editors. Data flows technical-to-business: Glue catalogs the bytes, DataZone makes them discoverable and governs who subscribes.

Crawlers, the Data Catalog, and partition tools answer where the data is and what its schema looks like. Deciding who may read which rows or columns is fine-grained access control, handled by Lake Formation, which is an authorization concern rather than a cataloging one. On a cataloging question, do not let a Lake Formation permissions distractor pull you off the metadata mechanics.

Set enableUpdateCatalog to true and updateBehavior to UPDATE_IN_DATABASE when a Glue ETL job writes a DynamicFrame to a catalog table; the job then overwrites the schema and registers new partitions during the run, so new data is immediately queryable in Athena without a separate crawler.

Trap Scheduling a crawler after every ETL run to pick up partitions the job could have registered itself.

Setting the crawler recrawl policy to CRAWL_NEW_FOLDERS_ONLY makes the first run a full scan and later runs process only new folders, cutting crawl time and cost for stable-schema tables that just gain partitions. This mode forces UpdateBehavior and DeleteBehavior to LOG, so a major schema change needs a temporary switch back to CRAWL_EVERYTHING.

SchemaChangePolicy UpdateBehavior=LOG records detected schema changes without overwriting an existing table definition, while CrawlerOutput AddOrUpdateBehavior=MergeNewColumns adds only newly discovered columns and keeps manual edits. DeleteBehavior=DEPRECATE_IN_DATABASE marks missing tables deprecated, and Partitions AddOrUpdateBehavior=InheritFromTable makes partitions adopt the table schema to avoid HIVE_PARTITION_SCHEMA_MISMATCH.

By default UpdateTable always creates an archived version of the table before applying changes, and GetTableVersions / GetTableVersion retrieve that history for auditing and schema comparison. Pass SkipArchive=true to suppress version creation for routine metadata edits; crawlers also create versions when they apply a schema change.

Trap Building a custom change-log pipeline when the catalog already keeps versioned table history.

A Lake Formation data filter combines a row filter expression (for example product_type='pharma') with a column include or exclude list, and you grant SELECT with that filter to give cell-level security. Specifying all columns plus a row expression yields row-level only; including/excluding columns plus all rows yields column-level only; combining both yields cell-level. Filters apply to read operations, so only SELECT can carry a filter.

Hybrid access mode lets existing IAM-based pipelines keep their access while you opt specific principals into Lake Formation fine-grained permissions on the same registered S3 location. It is the standard way to onboard new analyst teams to Lake Formation read access without disrupting existing ETL write workloads.

Lake Formation tag-based access control attaches LF-Tags (such as classification and domain) to databases and tables, then grants permissions via LF-Tag expressions. New resources tagged the same way are automatically covered, minimizing individual grants; cross-account LF-Tag sharing needs DESCRIBE and ASSOCIATE on the tags plus cross-account version 3 or higher.

Trap Granting table-by-table permissions to every new principal instead of matching on tags.

To create a Data Catalog table that points at an S3 location registered with Lake Formation, a principal needs both CREATE_TABLE on the database and DATA_LOCATION_ACCESS on that specific registered S3 location (or prefix). Registering the S3 location with a service-linked or custom IAM role is the prerequisite for any Lake Formation fine-grained access.

Amazon SageMaker Catalog, built on the former Amazon DataZone, catalogs data and machine-learning assets and traces their lineage so teams can discover, govern, and understand the origin of assets across the platform. Where ML Lineage Tracking captures a single workflow's provenance graph, the catalog is the broader discovery-and-governance layer over many assets.

Amazon Athena is an interactive, serverless query service that runs standard SQL (its engine is Trino/Presto) directly against data in S3, with no servers to manage and no step to aggregate or load the data first. That makes it the right call for one-off exploration, querying data-lake files, and troubleshooting logs. Reach for a data warehouse instead only when the same heavy analytics repeat on loaded, structured data.

Trap Loading the S3 data into a Redshift cluster first just to run a single exploratory query, which adds an ETL step Athena is designed to skip.

Amazon Redshift is a data warehouse, and AWS positions it as the best choice when you pull data from many sources into one structured model and run sophisticated reports whose queries join large numbers of very large tables. Its engine is optimized for exactly those repeated complex joins, which is where it beats querying raw files in place. The deciding signal is the recurring, highly-structured workload, not the raw size of the data.

Trap Picking Redshift because the dataset is large; size alone does not decide it, an infrequent ad hoc query on S3 still favors Athena.

Athena charges $5 per TB of data scanned per query, so cost tracks bytes read, not rows returned or query duration. Every cost-optimization answer for Athena reduces to reading fewer bytes. That is why storage format and layout, not query tuning alone, are the levers the exam tests.

Trap Assuming faster or shorter queries cost less; a quick query that still scans a full table is billed for the full scan.

Storing data as columnar Parquet or ORC lets Athena read only the columns a query names (column pruning) and skip files using min/max statistics, instead of scanning entire row-based CSV or JSON files. Because Athena prices per byte scanned, this directly shrinks the bill on selective queries. Pair it with compression so each remaining file is smaller still.

Trap Leaving data as CSV/JSON and trying to save money by selecting fewer columns; a row format still reads every column off disk regardless of the SELECT list.

Partitioning data on a column you filter by, for example date, organizes it into separate S3 prefixes so a WHERE clause on the partition key skips every prefix the query does not touch (partition pruning). Combined with a columnar format this can turn a year-long scan into a few partitions read. The partition key should match how queries filter, or the pruning never kicks in.

Trap Partitioning on a high-cardinality column queries rarely filter on, which creates many tiny partitions and adds overhead without reducing bytes scanned.

A CREATE TABLE AS SELECT (CTAS) query writes the result of a SELECT to a new table in S3 in one step and can convert the output to Parquet or ORC and partition or bucket it as it writes. Run it once to produce a compact, columnar, partitioned dataset, then point all later queries at that smaller table for far fewer bytes scanned. INSERT INTO appends more results to a CTAS-created table for incremental, serverless ELT inside Athena.

Athena Federated Query uses a data source connector that runs as an AWS Lambda function to query non-S3 sources such as DynamoDB, RDS and other JDBC databases, or CloudWatch Logs, and join them with S3 data in one SQL statement. It extends Athena's reach without copying the source data into S3 first. Federated and other non-S3 connectors carry a 10 MB minimum data-scanned charge per query.

Trap Building a separate ETL job to copy DynamoDB into S3 just so Athena can read it, when a federated connector queries DynamoDB in place.

Athena offers a simplified notebook experience that runs Apache Spark with Python serverlessly, so you can explore data interactively without sizing or managing a cluster. It is billed at $0.35 per DPU-hour for driver and worker compute. Reach for it when analysis is iterative or needs full Spark rather than the SQL that standard Athena queries provide.

Athena, Redshift Serverless, and provisioned Redshift run the same SQL; the difference is capacity management and billing. Serverless services (Athena per TB scanned, Redshift Serverless per RPU-second) remove cluster management and charge by use, which wins for variable, unpredictable, or intermittent workloads. A provisioned, reserved cluster gives the lowest rate per unit of work for steady, always-on load at the cost of sizing and managing it.

Trap Defaulting to a provisioned cluster for a bursty, occasionally-used workload, where it sits idle and costs more than a serverless option that bills only when in use.

Redshift Serverless automatically provisions data-warehouse capacity and scales it in seconds, measured in Redshift Processing Units (RPUs), and you pay only when the warehouse is in use. It gives the same Redshift SQL and performance without setting up, tuning, or managing a cluster, which fits unpredictable or volatile workloads. Choose provisioned Redshift instead when the load is steady enough to size and reserve a cluster for a lower steady-state rate.

Amazon QuickSight is the managed business intelligence service for building dashboards, charts, and business reports on top of analyzed data, and it also offers ML Insights such as anomaly detection and forecasting. When a question is about visualizing results or building a dashboard rather than running SQL, the answer is QuickSight, not a query engine. It is now delivered as Amazon Quick Sight within Amazon Quick, with existing QuickSight APIs unchanged.

Trap Answering a visualization or dashboard question with Athena or Redshift, which run the SQL behind a dashboard but are not the BI presentation layer.

QuickSight reads a dataset either by direct query, running SQL against the source on every view, or by importing it into SPICE (Super-fast, Parallel, In-memory Calculation Engine), its in-memory engine. SPICE makes dashboards respond fast and hits the source only on refresh rather than on every view, so it also limits cost against per-query sources like Athena. Use direct query when viewers must always see live source data; use SPICE for fast, frequently-viewed dashboards that tolerate scheduled freshness.

Trap Using direct query against Athena for a heavily-viewed dashboard, so every view re-scans S3 and re-incurs the $5/TB charge that a SPICE import would pay only on refresh.

Aggregation collapses many rows into one value with functions like SUM, COUNT, and AVG, and GROUP BY runs that aggregation once per group, for example total sales per region. Every non-aggregated column in the SELECT must appear in the GROUP BY. This is the building block behind most summary reports, and it runs identically in Athena and Redshift because both speak standard SQL.

A rolling (moving) average smooths a series over a sliding span and is expressed with a window function, such as AVG(amount) OVER (ORDER BY day ROWS BETWEEN 6 PRECEDING AND CURRENT ROW). Unlike GROUP BY, a window function keeps every row and adds the windowed value alongside it, which is exactly what a per-row trailing average needs. Aggregating with GROUP BY instead would collapse the rows and lose the per-day series.

Trap Reaching for GROUP BY to compute a rolling average; it collapses rows into one value per group, so the per-row moving series disappears.

Pivoting reshapes data from long to wide, turning distinct row values into separate columns to make a cross-tabulation, for example one column per month. In QuickSight a pivot table does this visually, and in SQL you express it with conditional aggregation (SUM with a CASE per target column). It is a presentation reshape, not a new computation, so the underlying numbers come from the same aggregation.

Amazon SageMaker Data Wrangler provides a visual data-flow with built-in transforms and data-quality insights to import, prepare, transform, and featurize data specifically for machine learning, importing from S3, Athena, Redshift, and other sources with little to no code. It is the ML-oriented prep tool and is now part of SageMaker Canvas. Choose it when the goal is feature engineering for a model, not general analyst cleanup.

EMR managed scaling resizes a cluster between minimum and maximum capacity units by sampling YARN workload metrics, requiring no hand-written scaling policies or workload prediction. Keeping Spark dynamic resource allocation (the default) enabled lets executors scale with the managed cluster; pairing Spot task nodes with On-Demand core nodes adds cost savings while protecting HDFS.

Trap Putting Spot Instances on core nodes; an interruption there loses HDFS data, so reserve Spot for task nodes.

The off-heap memory holding Java NIO buffers, thread stacks, and native libraries is spark.executor.memoryOverhead, which defaults to about 10% of executor memory (minimum 384 MB) and is heavily used during shuffles. When containers are killed for exceeding YARN memory, raising memoryOverhead is the first fix, not raising spark.executor.memory.

Trap Only increasing spark.executor.memory; that grows the heap but leaves off-heap shuffle memory short, so the kills continue.

Setting maximizeResourceAllocation to true in the spark configuration classification makes EMR compute the maximum compute and memory for one executor on a core-group instance and set spark-defaults accordingly. It is the low-effort choice for a Spark-only cluster on uniform instance groups.

Trap Using it with instance fleets of mixed sizes; it sizes from a single instance type, so set spark.executor settings manually for heterogeneous fleets.

A calculated field built in the dataset data-preparation phase that is row-level (no aggregation or parameters) is computed once and stored in SPICE during ingestion, removing the per-render cost. Move such ifelse or string-manipulation fields out of the analysis to speed up slow visuals on large datasets.

Trap Putting an aggregate- or parameter-based field in data prep expecting materialization; only non-aggregate row-level calculations are materialized, so leave those in the analysis.

Redshift does not analyze external tables, so without statistics it assumes an external table is larger than any local table and picks poor join strategies. Use ALTER TABLE to set the TABLE PROPERTIES numRows to the real row count so the optimizer can compare sizes and avoid broadcasting the wrong side.

Trap Running ANALYZE on the external table; Redshift cannot gather its statistics, so numRows must be set by hand.

Many tiny files hurt Redshift Spectrum because each query then scans far more, smaller objects with little parallelism benefit; AWS advises keeping file sizes larger than 64 MB and avoiding data-size skew by keeping files about the same size, so a query can be processed in parallel across evenly sized files. Converting to columnar Parquet lets Spectrum prune unneeded columns from the scan (column pruning) and apply predicate pushdown, and partitioning on a commonly filtered column enables partition pruning to limit data scanned.

Trap Only converting format while leaving thousands of tiny, skewed files; AWS still calls for files larger than 64 MB of roughly equal size, so query parallelism stays poor until the small files are merged.

A data service generates a one-time symmetric data key, encrypts your bytes locally with it, then asks AWS KMS to wrap that data key under a long-lived KMS key and stores the wrapped key beside the ciphertext. To read it back the service sends the wrapped data key to KMS, which unwraps it without the key material ever leaving KMS. This is why you almost never call KMS yourself for S3, Redshift, RDS, DynamoDB, or EBS; the service does it, and your only real choice is which KMS key wraps the data key.

An AWS owned key is shared across accounts, invisible to you, and free, used when a service encrypts by default. An AWS managed key (aws/s3, aws/redshift) lives in your account, is free to store, but its policy and rotation are fixed and it cannot be scoped or shared. A customer managed key (CMK) is yours: you write the key policy, share cross-account, control rotation, and every use is logged in CloudTrail. Reach for a CMK whenever the requirement names custom permissions, cross-account, auditing, or your own rotation schedule.

Trap Choosing an AWS managed key when the requirement is a custom key policy or cross-account sharing; its policy is fixed and AWS-controlled, so it cannot do either.

AWS managed keys rotate automatically once a year on an AWS-controlled schedule and you cannot change it; a customer managed key lets you enable automatic yearly rotation, trigger on-demand rotation, or turn rotation off. CloudTrail records every kms:Decrypt and kms:GenerateDataKey against your CMK, which is what "audit who used the key" requires. An AWS owned key produces no per-key visibility at all.

When KMS rotates a customer managed key it generates new backing material but keeps the same key ID, ARN, and alias, and retains the older material so ciphertext encrypted before rotation still decrypts. That means you do not re-encrypt existing data or update any reference when rotation happens. Rotation protects against long-term key-material exposure, not against a compromised key policy.

You cannot immediately delete a KMS key; you schedule deletion with a waiting period of 7 to 30 days (default 30), and you can cancel during the window. The window exists because once the key is gone every object it ever wrapped is permanently unreadable. Disable a key instead of deleting it when you only need to stop its use temporarily.

Trap Scheduling a key for deletion to quickly cut off access; the data wrapped by it becomes unrecoverable, so disable the key instead when the block is temporary.

AWS KMS HSMs are validated at FIPS 140-3 Security Level 3 in all Regions where KMS is offered, so a bare "needs FIPS 140-3 Level 3" requirement is satisfied by KMS alone. The CloudHSM trigger has shifted to single-tenant dedicated hardware or full customer custody of key material, not the FIPS level by itself.

Trap Selecting CloudHSM purely because a stem says "FIPS 140-3 Level 3"; KMS already meets that level, so the differentiator must be single-tenancy or full key custody.

A multi-Region key is a primary key plus replicas in other Regions that share the same key ID and key material, so data encrypted in one Region can be decrypted in another with the local replica. This is what cross-Region S3 replication of SSE-KMS objects or a DynamoDB global table needs; an ordinary single-Region key cannot decrypt material in a different Region. It is a deliberate choice at key creation, not the default.

Trap Assuming a normal customer managed key works for cross-Region replication; only a multi-Region key shares material across Regions, so a standard key blocks decryption at the destination.

All new S3 buckets encrypt objects at rest with S3-managed AES-256 keys (SSE-S3) by default, at no extra charge and with no per-bucket action. So "is this data encrypted at rest" is yes unless someone changed it. SSE-S3 gives no CloudTrail key auditing and no custom key policy; step up to SSE-KMS when you need those.

SSE-KMS wraps the data key under a KMS key, adding CloudTrail auditing, custom key policies, cross-account access, and rotation control. SSE-C (server-side encryption with customer-provided keys) means you send the key on every request and AWS uses then forgets it, so AWS never stores your key and losing it loses the object. DSSE-KMS applies dual-layer at-rest encryption. Match the stem: "audit and rotate" is SSE-KMS, "AWS must never hold the key" is SSE-C.

DSSE-KMS (dual-layer server-side encryption with KMS keys) applies two independent layers of AES-256: the first uses a KMS data key, the second uses a separate S3-managed key. It is not two KMS layers. Reach for it only when a requirement explicitly names dual-layer at-rest encryption; otherwise SSE-KMS is sufficient and cheaper, and DSSE-KMS does not support S3 Bucket Keys.

Trap Treating DSSE-KMS as two KMS layers or as the default for sensitive data; only its first layer is KMS-backed and plain SSE-KMS covers ordinary needs at lower cost.

Plain SSE-KMS calls KMS once per object operation, so a high-request bucket racks up kms:GenerateDataKey and kms:Decrypt charges and can hit throttling. S3 Bucket Keys derives a short-lived bucket-level key from KMS and generates per-object data keys locally, reducing KMS request traffic and cost by up to roughly 99%. It is a one-flag change, works with SSE-KMS only (not DSSE-KMS), and is the go-to answer when a stem complains about KMS request charges.

Trap Switching from SSE-KMS to SSE-S3 just to cut KMS request cost and losing the audit and policy control; S3 Bucket Keys keeps SSE-KMS while removing most of the per-object KMS calls.

RDS and Aurora encryption of storage, automated backups, snapshots, and read replicas is a KMS-backed choice made at instance creation. To encrypt an existing unencrypted instance you snapshot it, copy the snapshot with encryption enabled, and restore from that encrypted copy. A read replica inherits the source's encryption state.

Trap Assuming you can toggle encryption on a running unencrypted RDS instance; you must restore an encrypted copy from a snapshot instead.

DynamoDB encrypts every table at rest by default and you cannot turn it off; the only decision is AWS owned (default, free), AWS managed, or a customer managed key. Pick a customer managed key when you need CloudTrail auditing of key use or a custom key policy on the table's data.

An encrypted EBS volume encrypts data at rest, all of its snapshots, and the I/O between the volume and the instance, all with a KMS key. An encrypted snapshot can only be restored to an encrypted volume, and copies of encrypted snapshots stay encrypted. Account-level "encryption by default" forces every new volume to be encrypted.

Encryption in transit is TLS on the wire to every endpoint, and you can require it (an S3 policy denying aws:SecureTransport=false, Redshift require_ssl, RDS rds.force_ssl). Client-side, or before-transit, encryption goes further: your code encrypts the data with a key you control before it travels, so only ciphertext crosses the network and lands in the store and AWS never sees plaintext. When a stem says AWS must never see the data, that is client-side encryption or SSE-C, not ordinary at-rest encryption.

Trap Answering server-side SSE-KMS for "AWS must never see our plaintext"; the service still decrypts server-side, so only client-side encryption or SSE-C keeps plaintext or the key out of AWS.

An IAM policy in another account grants nothing on a KMS key until that key's own key policy first allows the other account. So sharing encrypted data across accounts is two-sided: account A's key policy trusts account B's account principal, then account B's IAM role is allowed kms:Decrypt. When account B cannot read account A's SSE-KMS data, the missing piece is almost always account A's key policy, not B's IAM policy.

Trap Granting only an IAM policy in the consumer account and expecting cross-account decrypt to work; without an allow in the key's own policy KMS denies it.

A key-policy principal like arn:aws:iam::<account>:root trusts that entire account as a delegation target, letting the account's administrators in turn grant access to their principals. It does not mean the account's root user. Reading it as the root user is a classic distractor on cross-account encryption questions.

Trap Reading arn:aws:iam::<account>:root in a key policy as the root user; it designates the whole account as a trusted principal.

A grant is purely additive (it can allow, never deny), targets exactly one KMS key and one grantee principal, and may point at a key in another account, which fits a service that assumes a role per job. Grant changes are eventually consistent, so use the returned grant token to exercise a new grant immediately. The limit is 50,000 grants per KMS key.

Trap Expecting a grant to deny or restrict access; a grant only adds permissions, so you cannot use it to carve back what a key policy already allows.

Redshift dynamic data masking attaches masking policies to columns and applies them per role at query time, so the stored value never changes and one table serves full, partial, or fully-redacted views to different roles. It is the answer when authorized but lower-privilege users must see only part of a sensitive column on the same live table. Encryption cannot do this, because anyone with the decrypt path sees the raw value.

Trap Reaching for encryption to hide values from an authorized analyst; they hold the decrypt path, so use dynamic data masking to control what each role sees.

AWS Glue PII detection transforms scan columns during an ETL job and can redact, hash, or partially mask matched entities such as names, emails, and card numbers, writing a cleaned data set to the target. Use it to sanitize data before it lands in a shared zone. Glue DataBrew offers similar masking, hashing, and redaction steps for data-prep recipes.

Masking and tokenization are reversible through a privileged path: masking shows different fidelity per role on the same stored value, and tokenization swaps the value for a reversible token while the real value sits in a separate vault, preserving joins and uniqueness downstream. Anonymization is deliberately one-way, with no recoverable mapping, used when the original must never be recovered. Decide by reversibility before picking the technique.

Trap Anonymizing data the business still needs to re-identify later; anonymization is one-way, so use tokenization when downstream must recover the original through the vault.

A customer managed KMS key costs about $1 per key per month plus per-request charges (roughly $0.03 per 10,000 requests), while AWS managed and AWS owned keys have no storage fee. That cost is the trade for custom policy, auditing, and rotation control, so do not create a CMK when default encryption with a free key meets the requirement. On a high-request SSE-KMS bucket, S3 Bucket Keys is what keeps the per-request side of that bill down.

With SSE-KMS and a customer managed key, the side that writes data (a Kinesis producer, a Glue job uploading objects) needs kms:GenerateDataKey on the key, and the side that reads (a Kinesis consumer, a Glue job downloading objects) needs kms:Decrypt. Granting these only in the identity policy is not enough: the KMS key policy must also allow the role to use the key. The same kms:Decrypt requirement applies to a Secrets Manager secret encrypted with a customer managed key.

Trap Giving a producer kms:Decrypt and expecting writes to work, when the encrypt path actually requires kms:GenerateDataKey.

The Glue Detect PII transform takes a per-entity action, so one job can SHA256-hash emails for consistent joins while REDACT-ing names. PARTIAL_REDACT with numRightCharsToExclude keeps the last N characters (e.g. last four digits) and redactChar sets the mask. Cell-level detection scans each value, sourceColumns scopes actions to chosen columns, and outputColumnName writes detection metadata to a new column for audit.

Trap Using full REDACT when the requirement is to keep the last four digits, which needs PARTIAL_REDACT with numRightCharsToExclude.