Databricks Certified Data Engineer Associate — Question 117

A data engineer is designing a data pipeline. The source system generates files in a shared directory that is also used by other processes. As a result, the files should be kept as is and will accumulate in the directory. The data engineer needs to identify which files are new since the previous run in the pipeline, and set up the pipeline to only ingest those new files with each run.

Which of the following tools can the data engineer use to solve this problem?

Answer options

Correct answer: D

Explanation

The correct answer is D, Auto Loader, which is designed to efficiently ingest new files from a directory while leaving existing files intact. Unity Catalog, Delta Lake, and Databricks SQL do not specifically provide the functionality to automatically identify and ingest only new files from a continuously growing directory.