Databricks Certified Generative AI Engineer Associate — Question 1

A Generative Al Engineer has created a RAG application to look up answers to questions about a series of fantasy novels that are being asked on the author’s web forum. The fantasy novel texts are chunked and embedded into a vector store with metadata (page number, chapter number, book title), retrieved with the user’s query, and provided to an LLM for response generation. The Generative AI Engineer used their intuition to pick the chunking strategy and associated configurations but now wants to more methodically choose the best values.
Which TWO strategies should the Generative AI Engineer take to optimize their chunking strategy and parameters? (Choose two.)

Answer options

• A. Change embedding models and compare performance.
• B. Add a classifier for user queries that predicts which book will best contain the answer. Use this to filter retrieval.
• C. Choose an appropriate evaluation metric (such as recall or NDCG) and experiment with changes in the chunking strategy, such as splitting chunks by paragraphs or chapters. Choose the strategy that gives the best performance metric.
• D. Pass known questions and best answers to an LLM and instruct the LLM to provide the best token count. Use a summary statistic (mean, median, etc.) of the best token counts to choose chunk size.
• E. Create an LLM-as-a-judge metric to evaluate how well previous questions are answered by the most appropriate chunk. Optimize the chunking parameters based upon the values of the metric.

Correct answer: C, E

Explanation

Option C is correct because it focuses on evaluating different chunking strategies using performance metrics, which is essential for optimization. Option E is also valid as it introduces a metric based on LLM performance for evaluating chunk relevance. Options A and B do not directly address the optimization of chunking parameters, while Option D suggests a method that may not accurately reflect chunk performance related to retrieval.