Google Cloud Professional Data Engineer — Question 17

You are designing a data processing pipeline. The pipeline must be able to scale automatically as load increases. Messages must be processed at least once and must be ordered within windows of 1 hour. How should you design the solution?

Answer options

• A. Use Apache Kafka for message ingestion and use Cloud Dataproc for streaming analysis.
• B. Use Apache Kafka for message ingestion and use Cloud Dataflow for streaming analysis.
• C. Use Cloud Pub/Sub for message ingestion and Cloud Dataproc for streaming analysis.
• D. Use Cloud Pub/Sub for message ingestion and Cloud Dataflow for streaming analysis.

Correct answer: D

Explanation

The correct answer is D because Cloud Pub/Sub provides excellent support for scalable message ingestion with at-least-once delivery guarantees, and Cloud Dataflow offers powerful capabilities for processing streaming data in order. Options A and B use Apache Kafka, which may not meet the same ease of use or integration for the specified requirements, while option C does not leverage the optimal processing capabilities of Cloud Dataflow.