Databricks Certified Data Engineer Professional — Question 230

A Spark job is taking longer than expected. Using the Spark UI, a data engineer notes that the Min, Median, and Max Durations for tasks in a particular stage show the minimum and median time to complete a task as roughly the same, but the max duration for a task to be roughly 100 times as long as the minimum.
Which situation is causing increased duration of the overall job?

Answer options

• A. Task queueing resulting from improper thread pool assignment.
• B. Spill resulting from attached volume storage being too small.
• C. Network latency due to some cluster nodes being in different regions from the source data
• D. Skew caused by more data being assigned to a subset of spark-partitions.
• E. Credential validation errors while pulling data from an external system.

Correct answer: D

Explanation

The correct answer is D because skew occurs when a disproportionate amount of data is assigned to a few partitions, causing those tasks to take significantly longer to complete. The other options do not adequately explain the observed performance issue, as task queueing, spill, network latency, and credential validation typically affect all tasks rather than causing extreme differences in task completion times.