AWS Certified Data Engineer – Associate (DEA-C01) — Question 172

A company receives a data file from a partner each day in an Amazon S3 bucket. The company uses a daily AWS Glue extract, transform, and load (ETL) pipeline to clean and transform each data file. The output of the ETL pipeline is written to a CSV file named Daily.csv in a second S3 bucket.

Occasionally, the daily data file is empty or is missing values for required fields. When the file is missing data, the company can use the previous day’s CSV file.

A data engineer needs to ensure that the previous day's data file is overwritten only if the new daily file is complete and valid.

Which solution will meet these requirements with the LEAST effort?

Answer options

• A. Invoke an AWS Lambda function to check the file for missing data and to fill in missing values in required fields.
• B. Configure the AWS Glue ETL pipeline to use AWS Glue Data Quality rules. Develop rules in Data Quality Definition Language (DQDL) to check for missing values in required fields and empty files.
• C. Use AWS Glue Studio to change the code in the ETL pipeline to fill in any missing values in the required fields with the most common values for each field.
• D. Run a SQL query in Amazon Athena to read the CSV file and drop missing rows. Copy the corrected CSV file to the second S3 bucket.

Correct answer: B

Explanation

Option B is correct because configuring the AWS Glue ETL pipeline with Data Quality rules allows it to automatically validate the incoming data for completeness and required fields without additional manual intervention. Other options either require more complex implementations (A, C) or do not directly address the need for validation before overwriting the data (D).