Google Cloud Professional Data Engineer — Question 43

You need to store and analyze social media postings in Google BigQuery at a rate of 10,000 messages per minute in near real-time. Initially, design the application to use streaming inserts for individual postings. Your application also performs data aggregations right after the streaming inserts. You discover that the queries after streaming inserts do not exhibit strong consistency, and reports from the queries might miss in-flight data. How can you adjust your application design?

Answer options

• A. Re-write the application to load accumulated data every 2 minutes.
• B. Convert the streaming insert code to batch load for individual messages.
• C. Load the original message to Google Cloud SQL, and export the table every hour to BigQuery via streaming inserts.
• D. Estimate the average latency for data availability after streaming inserts, and always run queries after waiting twice as long.

Correct answer: D

Explanation

Option D is correct because it allows for the acknowledgment of data latency after streaming inserts, ensuring that queries capture all relevant data. Option A would not provide the consistency needed for real-time analysis. Option B eliminates the benefits of streaming inserts, which are essential for real-time processing. Option C introduces unnecessary complexity by adding an intermediary step with Google Cloud SQL, which doesn't solve the consistency issue after streaming inserts.