Domain 1 of 4 · Chapter 2 of 3

Transform data and engineer features

Unlock the complete study guide.

Bundled into the existing AWS Certified Machine Learning Engineer - Associate premium course — no separate purchase.

14-day money-back guarantee — no questions asked.

Included in this chapter:

  • Where transformation fits between data and a model
  • Choosing a batch transformation engine
  • Per-record stream transforms in Amazon Data Firehose
  • Stateful stream processing with Flink and Glue streaming
  • Engineering numeric and categorical features
  • Advanced text pre-processing techniques
  • Managing features in Amazon SageMaker Feature Store
  • Embedding models: turning text and images into vectors
  • Preparing documents for RAG: chunking and metadata
  • Masking, redacting, and anonymizing sensitive data
  • Preparing FM fine-tuning, CPT, and distillation data
  • Requirements matched to transformation choices

Batch transformation engines

Decision axisAWS Glue DataBrewSageMaker Data WranglerAWS Glue ETL jobsSpark on Amazon EMR
Who writes the transformationAn analyst, as point-and-click recipe steps with no codeAn ML engineer, as a low-code flow with optional custom Python transformsAn engineer, as a Spark script or a visual AWS Glue Studio jobAn engineer, as their own Spark application code
How you develop itInteractively in a project on a data sampleInteractively in a flow on a data sampleIn a script or the Glue Studio editorIn code you submit to the cluster
How it runs on the full datasetA recipe job applies a published recipe versionA Data Wrangler job runs as a SageMaker Processing jobOn the serverless Spark-based ETL engine, with no cluster to manageOn your existing EMR cluster, reading Amazon S3 directly
Typical scenario cueRepeatable visual cleanup without writing codeA feature flow exported to Pipelines, Feature Store, or an inference pipeline, among other destinationsScheduled serverless ETL that should process only new inputAn existing EMR environment and existing Spark code

Decision tree

Data arrives as a continuous stream?Needs state across events?windows, joins, aggregatesEngineer builds a Spark job?script or Glue StudioFeaturize for an ML workflow?typical fit, not an exclusive routeExisting EMR clusterand Spark code?Managed Servicefor Apache Flinkor AWS Gluestreaming ETL jobFirehose with aLambda transformSageMakerData WranglerAWS GlueDataBrewSpark onAmazon EMRAWS Glue ETLjobsWhichever engine: fit transformations once and reuse them at inferenceYes, streamNo, batchYes, statefulNo, per-recordNo, author on a sampleYes, Spark codeYes, featurizeNo, no-code cleanupYes, EMRNo, serverless

Cheat sheet

  • DataBrew recipes preserve an ordered sequence of preparation steps
  • A Data Wrangler flow must be executed on the full dataset for production output
  • Glue job bookmarks support incremental processing of supported sources
  • Rewinding a Glue bookmark does not remove existing target output
  • Glue bookmark pause processes data without advancing saved bookmark state
  • Spark on EMR supports distributed preprocessing in an existing batch environment
  • Inference preprocessing must reuse the transformations fitted during training
  • Feature metadata enables reuse without embedding sensitive values in descriptions
  • Feature Store TTL is based on event time and applies to online records
  • Record-level TTL overrides a feature group's default retention duration
  • A Firehose transformation must finish within Firehose's invocation limit
  • A Firehose transform must identify the disposition of each record
  • Firehose transformation batches must fit Lambda request and response limits
  • Glue streaming jobs use checkpoints for recovery rather than batch bookmarks
  • Windowed feature aggregation needs a stateful streaming processor
  • Firehose partition-key extraction needs Lambda when the payload is not directly parseable JSON
  • Flink checkpoints and snapshots preserve application state across different restart paths
  • Firehose must separate bundled events before dynamically partitioning them
  • Flink Studio suits interactive exploration while IDE development supports maintained streaming applications
  • Flink's DataStream API supports fine-grained state control and custom timers
  • Flink asynchronous I/O overlaps external enrichment requests instead of waiting for each response
  • Stable Flink operator UIDs preserve saved-state associations as application flow evolves
  • Tumbling windows form disjoint intervals while hopping windows can overlap
  • Unseen categories follow the fitted encoder's Invalid handling strategy
  • Min-max scaling and standardization produce different numeric representations
  • Robust scaling uses statistics resistant to outliers
  • A log transformation can compress a positive long-tailed feature
  • Binning represents a numeric measurement by its interval
  • One-hot encoding represents nominal categories without imposing a numeric ranking
  • Feature splitting exposes useful components of a compound value
  • Unit-vector normalization rescales each vector rather than each feature column
  • Custom tokenization can preserve domain-specific text distinctions
  • Term-frequency and document-frequency filters remove different kinds of text features
  • Embedding models produce numerical representations rather than generated answers
  • Query and document embeddings must use a compatible embedding space
  • Multimodal embeddings can place text and images in a shared retrieval space
  • Embedding input limits require segmenting oversized documents
  • TF-IDF combines local term frequency with corpus-wide term rarity
  • Feature hashing avoids a stored vocabulary but can collide terms
  • Knowledge-base CSV ingestion separates one content column from metadata columns
  • A higher semantic breakpoint threshold generally produces fewer larger chunks
  • Multimodal chunking depends on whether media is embedded directly or converted to text
  • Overlapping fixed-size chunks preserve context across chunk boundaries
  • Semantic chunking uses meaning to locate document boundaries
  • Hierarchical chunking combines focused matching with broader returned context
  • No chunking preserves each source document as a single retrieval unit
  • A knowledge-base data source's chunking strategy cannot be changed in place
  • Document metadata supplies retrieval filters separately from embedded content
  • Comprehend PII detection locates sensitive spans in text
  • PII redaction can conceal characters or replace spans with entity types
  • Deterministic encryption preserves equality patterns in protected columns
  • Reversible protection requires encryption rather than hashing or removal
  • Substitution and shuffling preserve different properties of sensitive data
  • Anonymization requires preventing likely identification rather than merely replacing identifiers
  • Fine-tuning demonstrations and continued-pretraining corpora serve different purposes
  • Bedrock customization data must satisfy both JSONL syntax and the model's schema
  • Training conversations should match the structure used at inference
  • Distillation prompts should represent the workload the student must perform
  • Multimodal fine-tuning examples must keep media linked to the correct conversation
  • Domain-specific text augmentation must preserve the intended task and answer

Unlock with Premium — includes all practice exams and the complete study guide.

References

  1. https://docs.aws.amazon.com/sagemaker/latest/dg/data-wrangler-transform.html
  2. https://docs.aws.amazon.com/aws-certification/latest/machine-learning-engineer-associate-02/machine-learning-engineer-associate-02-domain1.html
  3. https://docs.aws.amazon.com/sagemaker/latest/dg/data-wrangler-data-export.html
  4. https://docs.aws.amazon.com/sagemaker/latest/dg/inference-pipelines.html
  5. https://docs.aws.amazon.com/databrew/latest/dg/jobs.html
  6. https://docs.aws.amazon.com/databrew/latest/dg/what-is.html
  7. https://docs.aws.amazon.com/sagemaker/latest/dg/data-wrangler.html
  8. https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html
  9. https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark.html
  10. https://docs.aws.amazon.com/databrew/latest/dg/projects.html
  11. https://docs.aws.amazon.com/databrew/latest/dg/recipes.html
  12. https://docs.aws.amazon.com/glue/latest/dg/monitor-continuations.html
  13. https://docs.aws.amazon.com/glue/latest/dg/programming-etl-connect-bookmarks.html
  14. https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html
  15. https://docs.aws.amazon.com/firehose/latest/dev/create-name.html
  16. https://docs.aws.amazon.com/firehose/latest/dev/data-transformation-status-model.html
  17. https://docs.aws.amazon.com/firehose/latest/dev/dynamic-partitioning-partitioning-keys.html
  18. https://docs.aws.amazon.com/firehose/latest/dev/dynamic-partitioning-multirecord-deaggergation.html
  19. https://docs.aws.amazon.com/firehose/latest/dev/record-format-conversion.html
  20. https://docs.aws.amazon.com/managed-flink/latest/java/what-is.html
  21. https://docs.aws.amazon.com/managed-flink/latest/java/how-async.html
  22. https://docs.aws.amazon.com/managed-flink/latest/java/best-practices.html
  23. https://docs.aws.amazon.com/managed-flink/latest/java/how-snapshots.html
  24. https://docs.aws.amazon.com/kinesisanalytics/latest/dev/tumbling-window-concepts.html
  25. https://docs.aws.amazon.com/managed-flink/latest/java/how-zeppelin-sql-examples.html
  26. https://aws.amazon.com/blogs/big-data/top-10-flink-sql-queries-to-try-in-amazon-kinesis-data-analytics-studio/
  27. https://docs.aws.amazon.com/kinesisanalytics/latest/dev/timestamps-rowtime-concepts.html
  28. https://docs.aws.amazon.com/managed-flink/latest/java/how-fault.html
  29. https://docs.aws.amazon.com/glue/latest/dg/add-job-streaming.html
  30. https://docs.aws.amazon.com/databrew/latest/dg/recipe-actions.data-science.html
  31. https://docs.aws.amazon.com/databrew/latest/dg/recipe-actions.SKEWNESS.html
  32. https://docs.aws.amazon.com/redshift/latest/dg/r_LOG.html
  33. https://docs.aws.amazon.com/databrew/latest/dg/recipe-actions.BUCKETIZATION.html
  34. https://docs.aws.amazon.com/nova/latest/nova2-userguide/nova-data-prep-sft-2.html
  35. https://docs.aws.amazon.com/sagemaker/latest/dg/feature-store.html
  36. https://docs.aws.amazon.com/sagemaker/latest/dg/feature-store-concepts.html
  37. https://docs.aws.amazon.com/sagemaker/latest/dg/feature-store-time-to-live.html
  38. https://docs.aws.amazon.com/bedrock/latest/userguide/titan-embedding-models.html
  39. https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-titan-embed-text.html
  40. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/VectorSearchBestPractices.html
  41. https://docs.aws.amazon.com/bedrock/latest/userguide/kb-how-it-works.html
  42. https://docs.aws.amazon.com/bedrock/latest/userguide/titan-multiemb-models.html
  43. https://docs.aws.amazon.com/bedrock/latest/userguide/kb-chunking.html
  44. https://docs.aws.amazon.com/bedrock/latest/userguide/s3-data-source-connector.html
  45. https://docs.aws.amazon.com/bedrock/latest/userguide/kb-ds-update.html
  46. https://docs.aws.amazon.com/bedrock/latest/userguide/kb-metadata.html
  47. https://docs.aws.amazon.com/aws-certification/latest/machine-learning-engineer-associate-02/machine-learning-engineer-associate-02-domain4.html
  48. https://docs.aws.amazon.com/comprehend/latest/dg/how-pii.html
  49. https://docs.aws.amazon.com/comprehend/latest/dg/redact-api-pii.html
  50. https://docs.aws.amazon.com/databrew/latest/dg/personal-information-protection.html
  51. https://docs.aws.amazon.com/wellarchitected/latest/analytics-lens/best-practice-3.1-privacy-by-design..html
  52. https://docs.aws.amazon.com/aws-certification/latest/machine-learning-engineer-associate-02/machine-learning-engineer-associate-02-domain2.html
  53. https://docs.aws.amazon.com/bedrock/latest/userguide/model-customization-prepare.html
  54. https://docs.aws.amazon.com/nova/latest/nova2-userguide/nova-cpt.html
  55. https://docs.aws.amazon.com/bedrock/latest/userguide/model-distillation.html