Databricks Certified Associate Developer for Apache Spark — Question 3

Of the following situations, in which will it be most advantageous to store DataFrame df at the MEMORY_AND_DISK storage level rather than the MEMORY_ONLY storage level?

Answer options

• A. When all of the computed data in DataFrame df can fit into memory.
• B. When the memory is full and it’s faster to recompute all the data in DataFrame df rather than read it from disk.
• C. When it’s faster to recompute all the data in DataFrame df that cannot fit into memory based on its logical plan rather than read it from disk.
• D. When it’s faster to read all the computed data in DataFrame df that cannot fit into memory from disk rather than recompute it based on its logical plan.
• E. The storage level MENORY_ONLY will always be more advantageous because it’s faster to read data from memory than it is to read data from disk.

Correct answer: D

Explanation

The correct answer is D because when data cannot fit into memory, it can be more efficient to read it from disk rather than recompute it, especially if the logical plan for recomputation is complex. Options A, B, and C do not support using MEMORY_AND_DISK since they either imply full memory usage or faster recomputation. Option E is incorrect as it ignores scenarios where data may not fit into memory.