Google Cloud Professional Data Engineer — Question 97

You have developed three data processing jobs. One executes a Cloud Dataflow pipeline that transforms data uploaded to Cloud Storage and writes results to
BigQuery. The second ingests data from on-premises servers and uploads it to Cloud Storage. The third is a Cloud Dataflow pipeline that gets information from third-party data providers and uploads the information to Cloud Storage. You need to be able to schedule and monitor the execution of these three workflows and manually execute them when needed. What should you do?

Answer options

• A. Create a Direct Acyclic Graph in Cloud Composer to schedule and monitor the jobs.
• B. Use Stackdriver Monitoring and set up an alert with a Webhook notification to trigger the jobs.
• C. Develop an App Engine application to schedule and request the status of the jobs using GCP API calls.
• D. Set up cron jobs in a Compute Engine instance to schedule and monitor the pipelines using GCP API calls.

Correct answer: A

Explanation

The correct answer is A, as creating a Direct Acyclic Graph in Cloud Composer provides the necessary tools for scheduling and monitoring complex workflows effectively. Option B, while it involves monitoring, does not directly provide scheduling capabilities for the jobs. Option C lacks the integrated scheduling and monitoring features that Cloud Composer offers, and option D introduces unnecessary complexity with Compute Engine instead of leveraging managed services.