Data Engineering
Batch Processing & Spark
6 practice questions. Free questions open a full answer guide; the rest unlock with Pro.
- In Spark, what's the difference between a narrow and a wide transformation, and why does the distinction matter for performance?
- A nightly Spark job aggregates a large fact table and a teammate suggests just bumping spark.sql.shuffle.partitions and the cluster size until it's fast. How do you decide what's actually limiting it, and when is Spark even the right tool over a warehouse SQL query?Go Pro
- Your Spark pipeline writes correct results, but the output is thousands of tiny files and downstream reads have become painfully slow. What's going wrong and how would you fix the write?Go Pro
- Your Spark job writes a partitioned table and downstream queries have gotten slow because each daily partition contains thousands of tiny files. How does this happen, and how do you fix the write so it produces well-sized files without hurting the job?Go Pro
- What is data skew in a Spark job, how would you spot it, and what can you do about it?Go Pro
- Your Spark job is dominated by one or two tasks that run far longer than the rest during a join or aggregation. How would you diagnose the cause and fix it?Go Pro
Want questions matched to your role? Paste a job title, job description, or CV for a personalized set, or go Pro to unlock the full bank.