Transform data and engineer features
Unlock the complete study guide.
Bundled into the existing AWS Certified Machine Learning Engineer - Associate premium course — no separate purchase.
14-day money-back guarantee — no questions asked.
Included in this chapter:
- Where transformation fits between data and a model
- Choosing a batch transformation engine
- Per-record stream transforms in Amazon Data Firehose
- Stateful stream processing with Flink and Glue streaming
- Engineering numeric and categorical features
- Advanced text pre-processing techniques
- Managing features in Amazon SageMaker Feature Store
- Embedding models: turning text and images into vectors
- Preparing documents for RAG: chunking and metadata
- Masking, redacting, and anonymizing sensitive data
- Preparing FM fine-tuning, CPT, and distillation data
- Requirements matched to transformation choices
Batch transformation engines
| Decision axis | AWS Glue DataBrew | SageMaker Data Wrangler | AWS Glue ETL jobs | Spark on Amazon EMR |
|---|---|---|---|---|
| Who writes the transformation | An analyst, as point-and-click recipe steps with no code | An ML engineer, as a low-code flow with optional custom Python transforms | An engineer, as a Spark script or a visual AWS Glue Studio job | An engineer, as their own Spark application code |
| How you develop it | Interactively in a project on a data sample | Interactively in a flow on a data sample | In a script or the Glue Studio editor | In code you submit to the cluster |
| How it runs on the full dataset | A recipe job applies a published recipe version | A Data Wrangler job runs as a SageMaker Processing job | On the serverless Spark-based ETL engine, with no cluster to manage | On your existing EMR cluster, reading Amazon S3 directly |
| Typical scenario cue | Repeatable visual cleanup without writing code | A feature flow exported to Pipelines, Feature Store, or an inference pipeline, among other destinations | Scheduled serverless ETL that should process only new input | An existing EMR environment and existing Spark code |
Decision tree
Cheat sheet
Unlock with Premium — includes all practice exams and the complete study guide.
References
- https://docs.aws.amazon.com/sagemaker/latest/dg/data-wrangler-transform.html
- https://docs.aws.amazon.com/aws-certification/latest/machine-learning-engineer-associate-02/machine-learning-engineer-associate-02-domain1.html
- https://docs.aws.amazon.com/sagemaker/latest/dg/data-wrangler-data-export.html
- https://docs.aws.amazon.com/sagemaker/latest/dg/inference-pipelines.html
- https://docs.aws.amazon.com/databrew/latest/dg/jobs.html
- https://docs.aws.amazon.com/databrew/latest/dg/what-is.html
- https://docs.aws.amazon.com/sagemaker/latest/dg/data-wrangler.html
- https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html
- https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark.html
- https://docs.aws.amazon.com/databrew/latest/dg/projects.html
- https://docs.aws.amazon.com/databrew/latest/dg/recipes.html
- https://docs.aws.amazon.com/glue/latest/dg/monitor-continuations.html
- https://docs.aws.amazon.com/glue/latest/dg/programming-etl-connect-bookmarks.html
- https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html
- https://docs.aws.amazon.com/firehose/latest/dev/create-name.html
- https://docs.aws.amazon.com/firehose/latest/dev/data-transformation-status-model.html
- https://docs.aws.amazon.com/firehose/latest/dev/dynamic-partitioning-partitioning-keys.html
- https://docs.aws.amazon.com/firehose/latest/dev/dynamic-partitioning-multirecord-deaggergation.html
- https://docs.aws.amazon.com/firehose/latest/dev/record-format-conversion.html
- https://docs.aws.amazon.com/managed-flink/latest/java/what-is.html
- https://docs.aws.amazon.com/managed-flink/latest/java/how-async.html
- https://docs.aws.amazon.com/managed-flink/latest/java/best-practices.html
- https://docs.aws.amazon.com/managed-flink/latest/java/how-snapshots.html
- https://docs.aws.amazon.com/kinesisanalytics/latest/dev/tumbling-window-concepts.html
- https://docs.aws.amazon.com/managed-flink/latest/java/how-zeppelin-sql-examples.html
- https://aws.amazon.com/blogs/big-data/top-10-flink-sql-queries-to-try-in-amazon-kinesis-data-analytics-studio/
- https://docs.aws.amazon.com/kinesisanalytics/latest/dev/timestamps-rowtime-concepts.html
- https://docs.aws.amazon.com/managed-flink/latest/java/how-fault.html
- https://docs.aws.amazon.com/glue/latest/dg/add-job-streaming.html
- https://docs.aws.amazon.com/databrew/latest/dg/recipe-actions.data-science.html
- https://docs.aws.amazon.com/databrew/latest/dg/recipe-actions.SKEWNESS.html
- https://docs.aws.amazon.com/redshift/latest/dg/r_LOG.html
- https://docs.aws.amazon.com/databrew/latest/dg/recipe-actions.BUCKETIZATION.html
- https://docs.aws.amazon.com/nova/latest/nova2-userguide/nova-data-prep-sft-2.html
- https://docs.aws.amazon.com/sagemaker/latest/dg/feature-store.html
- https://docs.aws.amazon.com/sagemaker/latest/dg/feature-store-concepts.html
- https://docs.aws.amazon.com/sagemaker/latest/dg/feature-store-time-to-live.html
- https://docs.aws.amazon.com/bedrock/latest/userguide/titan-embedding-models.html
- https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-titan-embed-text.html
- https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/VectorSearchBestPractices.html
- https://docs.aws.amazon.com/bedrock/latest/userguide/kb-how-it-works.html
- https://docs.aws.amazon.com/bedrock/latest/userguide/titan-multiemb-models.html
- https://docs.aws.amazon.com/bedrock/latest/userguide/kb-chunking.html
- https://docs.aws.amazon.com/bedrock/latest/userguide/s3-data-source-connector.html
- https://docs.aws.amazon.com/bedrock/latest/userguide/kb-ds-update.html
- https://docs.aws.amazon.com/bedrock/latest/userguide/kb-metadata.html
- https://docs.aws.amazon.com/aws-certification/latest/machine-learning-engineer-associate-02/machine-learning-engineer-associate-02-domain4.html
- https://docs.aws.amazon.com/comprehend/latest/dg/how-pii.html
- https://docs.aws.amazon.com/comprehend/latest/dg/redact-api-pii.html
- https://docs.aws.amazon.com/databrew/latest/dg/personal-information-protection.html
- https://docs.aws.amazon.com/wellarchitected/latest/analytics-lens/best-practice-3.1-privacy-by-design..html
- https://docs.aws.amazon.com/aws-certification/latest/machine-learning-engineer-associate-02/machine-learning-engineer-associate-02-domain2.html
- https://docs.aws.amazon.com/bedrock/latest/userguide/model-customization-prepare.html
- https://docs.aws.amazon.com/nova/latest/nova2-userguide/nova-cpt.html
- https://docs.aws.amazon.com/bedrock/latest/userguide/model-distillation.html