Databricks Certified Data Engineer Professional — Question 45
The data science team has requested assistance in accelerating queries on free form text from user reviews. The data is currently stored in Parquet with the below schema:
item_id INT, user_id INT, review_id INT, rating FLOAT, review STRING
The review column contains the full text of the review left by the user. Specifically, the data science team is looking to identify if any of 30 key words exist in this field.
A junior data engineer suggests converting this data to Delta Lake will improve query performance.
Which response to the junior data engineer s suggestion is correct?
Answer options
- A. Delta Lake statistics are not optimized for free text fields with high cardinality.
- B. Text data cannot be stored with Delta Lake.
- C. ZORDER ON review will need to be run to see performance gains.
- D. The Delta log creates a term matrix for free text fields to support selective filtering.
- E. Delta Lake statistics are only collected on the first 4 columns in a table.
Correct answer: A
Explanation
The correct answer is A because Delta Lake statistics are not designed to handle free text fields that have high cardinality, which can lead to inefficient query performance. Options B and D are incorrect as Delta Lake can indeed store text data, and it does not create a term matrix for selective filtering. Option C is misleading because while ZORDER can help, it does not guarantee performance gains in this context, and E is false since Delta Lake can collect statistics on all columns.