AWS Certified Data Engineer – Associate (DEA-C01) — Question 224
A company uses Amazon S3 and AWS Glue Data Catalog to manage a data lake that contains contact information for customers. The company uses PySpark and AWS Glue jobs with a DynamicFrame to run a workflow that processes data within the data lake.
A data engineer notices that the workflow is generating errors as a result of how customer postal codes are stored in the data lake. Some postal codes include unnecessary numbers or invalid characters.
The data engineer needs a solution to address the errors and correct the postal codes in the data lake.
Answer options
- A. Create a schema definition for PySpark that matches the format the processing workflow requires for postal codes. Pass the schema to the DynamicFrame during processing.
- B. Use AWS Glue workflow properties to allow job state sharing. Configure the AWS Glue jobs to read values from the postal code column by using the properties from a previously successful run of the jobs.
- C. Configure the column.push_down_predicate setting and the catalogPartitionPredicate settings for the postal code column in the DynamicFrame.
- D. Set the DynamicFrame additional_options parameter ‘useS3ListImplementation’ to True.
Correct answer: A
Explanation
The correct answer is A because creating a schema definition that matches the expected format for postal codes ensures that the data is processed correctly without errors. The other options do not directly address the need to format or clean the postal codes, as B focuses on job state sharing, C on predicate settings, and D on implementation options that do not resolve the formatting issue.