Databricks Certified Generative AI Engineer Associate — Question 11

A Generative AI Engineer just deployed an LLM application at a digital marketing company that assists with answering customer service inquiries.
Which metric should they monitor for their customer service LLM application in production?

Answer options

• A. Number of customer inquiries processed per unit of time
• B. Energy usage per query
• C. Final perplexity scores for the training of the model
• D. HuggingFace Leaderboard values for the base LLM

Correct answer: A

Explanation

The correct answer is A because monitoring the number of customer inquiries processed per unit of time provides insight into the application's efficiency and responsiveness in real-time customer service scenarios. Options B, C, and D are less relevant for assessing the application's operational performance; energy usage per query does not directly indicate effectiveness, perplexity scores pertain to training rather than live performance, and leaderboard values are not applicable to a specific production environment.