Google Cloud Professional Machine Learning Engineer — Question 295

Your team is experimenting with developing smaller, distilled LLMs for a specific domain. You have performed batch inference on a dataset by using several variations of your distilled LLMs and stored the batch inference outputs in Cloud Storage. You need to create an evaluation workflow that integrates with your existing Vertex AI pipeline to assess the performance of the LLM versions while also tracking artifacts. What should you do?

Answer options

• A. Develop a custom Python component that reads the batch inference outputs from Cloud Storage, calculates evaluation metrics, and writes the results to a BigQuery table.
• B. Use a Dataflow component that processes the batch inference outputs from Cloud Storage, calculates evaluation metrics in a distributed manner, and writes the results to a BigQuery table.
• C. Create a custom Vertex AI Pipelines component that reads the batch inference outputs from Cloud Storage, calculates evaluation metrics, and writes the results to a BigQuery table.
• D. Use the Automatic side-by-side (AutoSxS) pipeline component that processes the batch inference outputs from Cloud Storage, aggregates evaluation metrics, and writes the results to a BigQuery table.

Correct answer: D

Explanation

The correct answer is D because the Automatic side-by-side (AutoSxS) pipeline component is specifically designed to evaluate different model versions in a streamlined manner while tracking performance metrics. Options A and C involve creating custom components, which may not integrate as seamlessly with Vertex AI's tracking capabilities. Option B, while useful for distributed processing, may not align as effectively with the existing Vertex AI pipeline structure for artifact tracking.