In Spark, what's the difference between a narrow and a wide transformation, and why does the distinction matter for performance?

technical-conceptual · Junior level · data-ml

What the interviewer is really asking

Assess whether the candidate understands that wide transformations trigger a shuffle and can connect that to job performance, rather than treating all Spark operations as equally cheap.

What to say

• Define both: a narrow transformation (filter, select, map, withColumn) needs only data within a single partition, so it runs without moving data across the cluster; a wide transformation (groupBy, join, distinct, orderBy) needs data from many partitions, which forces a shuffle.
• Explain why the shuffle is the cost: it writes intermediate data to disk and sends it over the network to regroup it, so it's the dominant expense in most Spark jobs and is the boundary that splits a job into stages.
• Give the practical lever: filter early to shrink the data before any wide step, broadcast a small table to avoid a shuffle join, and prefer reduceByKey over groupByKey because it reduces map-side before shuffling.

What to avoid

• Claiming all Spark transformations cost roughly the same, or that you can't influence shuffle cost.
• Confusing transformations with actions — transformations are lazy and only run when an action like count or write triggers them.
• Ordering or joining the full dataset first and filtering afterwards, which shuffles far more data than necessary.

Example answers

Strong: On a job that joined a 2-billion-row events table to a small 5k-row lookup, I broadcast the lookup so the join stayed narrow instead of shuffling the big table — the stage that had been the bottleneck dropped from minutes to seconds.

Weak: Narrow and wide are just two ways to write the same thing, so I use whichever reads more cleanly.