Previous version: MLA-C01. The current version is
MLA-C02.
Choosing a data-transformation tool on AWS
| Dimension | Data Wrangler | Glue DataBrew | AWS Glue (Spark) | Spark on EMR |
|---|
| Interface | Visual flow in SageMaker Canvas | No-code visual recipes | Visual (Glue Studio) or Spark code | Spark code on a cluster |
| Primary user | Data scientist doing EDA | Analyst cleaning data | Data engineer building ETL | Engineer needing cluster control |
| Scale | Sampled flows, export to job | Terabytes, serverless | Any size, serverless Spark | Very large, tunable cluster |
| Coding | Little to none, custom PySpark optional | None | PySpark or Scala | Full Spark control |
| Best for | Explore then export to pipeline/Feature Store | Repeatable no-code cleaning | Scheduled large scripted ETL | Custom or very large transforms |
| Catalog/recipe reuse | Reusable .flow file | Reusable recipe | Glue Data Catalog + jobs | Self-managed |
Decision tree
Cheat sheet Cheat sheet
- Use Data Wrangler for visual EDA that exports into a pipeline
- Use Glue DataBrew for no-code cleaning with reusable recipes
- Use AWS Glue for serverless, scripted, repeatable Spark ETL
- Choose EMR over Glue only when you need cluster control
- Glue FindMatches deduplicates records that are imperfect matches
- One-hot encode low-cardinality nominal categories
- Use ordinal or label encoding only when a real rank exists
- High-cardinality categoricals need binary, hashing, or embeddings
- Standardize with the mean and variance, normalize to a fixed range
- Skip scaling for tree-based models
- Impute missing values with a statistic learned from training data
- Detect outliers with IQR or standard-deviation rules, then cap or transform
- Feature splitting and binning expose hidden structure
- Feature Store online store serves the latest record at ms latency
- Feature Store offline store keeps full history in S3 for training
- Write both Feature Store stores to kill train-serve skew
- A feature group needs a record identifier and an event time
- Use Ground Truth to label data, choosing one of three workforces
- Ground Truth automated data labeling uses active learning to cut cost
- Transform streaming records with Lambda for per-record work
- Data Wrangler can export an engineered flow straight into Feature Store
- SageMaker Processing reads and writes only under /opt/ml/processing/
- SageMaker Processing exposes job and cluster config in /opt/ml/config JSON files
- ShardedByS3Key splits objects across instances; FullyReplicated copies all data to each
- Use a framework processor for built-in libraries and ScriptProcessor for a custom container
- Glue job bookmarks with transformation_ctx process only new data incrementally
- Glue DynamicFrames encode mixed-type fields as choice types, resolved with resolveChoice
- Glue ETL updates the Data Catalog itself with enableUpdateCatalog + UPDATE_IN_DATABASE
- Use Apache Iceberg for the Feature Store offline store to get time-travel and compaction
- Backfill the offline store directly with the Feature Store Spark connector, target_stores=OfflineStore
- Data Wrangler custom transforms should use PySpark (not pandas) for large datasets
- Data Wrangler Quick Model rates predictive power; Target Leakage flags features unavailable at prediction time
- DataBrew recipes use named steps for imputation, scaling, and encoding