High-Performing Data Ingestion and Transformation

Bundled into the existing AWS Certified Solutions Architect – Associate premium course — no separate purchase.

Included in this chapter:

Service	Type	Latency	Best for
Kinesis Data Streams	Stream	Sub-second	Ordered events with replay (24h–365d); custom consumers
Kinesis Firehose	Stream → store	60+ s buffer	Fire-and-forget delivery to S3 / Redshift / OpenSearch / Splunk
MSK (Managed Kafka)	Stream	Sub-second	Kafka-compatible workloads; existing Kafka tooling
AWS Glue	Serverless ETL (Spark + Python)	Minutes	Recurring ETL; schema discovery via crawlers; Glue Data Catalog
EMR	Hadoop / Spark / Hive / Presto / Flink cluster	Minutes-hours	Large-scale data processing; non-Glue frameworks
Athena	Serverless SQL over S3	Seconds-minutes	Ad-hoc SQL; per-TB-scanned billing
Redshift	Columnar MPP DW	Seconds-minutes	Petabyte analytics warehouse; BI dashboards
Lake Formation	Data lake governance	—	Centralized fine-grained access on S3 + Glue Catalog
DataSync / DMS / Snowball	Transfer	Hours-days	On-prem or other cloud → S3 migration

Cheat sheet

Streaming vs batch — pick by latency requirement
Pick transformation tool by data shape + size
Kinesis Data Firehose buffers at time OR size threshold
Kinesis Data Streams: 1 shard = 1 MB/s in, 2 MB/s out
Glue Data Catalog is THE metastore for the lake
Lake Formation = fine-grained access control over the lake
Glue worker types: G.1X / G.2X / G.4X / G.8X / G.025X
DMS: replicate from source to target with optional CDC
Snowball Edge for offline bulk transfer; Snowmobile retired
IoT Core error action fires when the primary rule action fails — preserves messages
IoT Core Basic Ingest bypasses the message broker — no per-message cost
Firehose Lambda transform: return recordId + result status + base64 data

Unlock with Premium — includes all practice exams and the complete study guide.

Cheat sheet