Certified Generative AI Engineer Associate Exam (Databricks)
2026-2028 Latest Version: 6.0 questions with verified answers
and rationales | instant pdf download
Question: 1
A Generative Al Engineer has created a RAG application to look up answers to questions
about a series of fantasy novels that are being asked on the author’s web forum. The
fantasy novel texts are chunked and embedded into a vector store with metadata (page
number, chapter number, book title), retrieved with the user’s query, and provided to an
LLM for response generation. The Generative AI Engineer used their intuition to pick the
chunking strategy and associated configurations but now wants to more methodically
choose the best values.
Which TWO strategies should the Generative AI Engineer take to optimize their chunking
strategy and parameters? (Choose two.)
A. Change embedding models and compare performance.
B. Add a classifier for user queries that predicts which book will best contain the answer.
Use this to filter retrieval.
C. Choose an appropriate evaluation metric (such as recall or NDCG) and experiment
with changes in the chunking strategy, such as splitting chunks by paragraphs or
chapters.
Choose the strategy that gives the best performance metric.
D. Pass known questions and best answers to an LLM and instruct the LLM to provide the
best token count. Use a summary statistic (mean, median, etc.) of the best token counts
to choose chunk size.
E. Create an LLM-as-a-judge metric to evaluate how well previous questions are
answered by the most appropriate chunk. Optimize the chunking parameters based
upon the values of the metric.
Answer: C, E
Explanation:
To optimize a chunking strategy for a Retrieval-Augmented Generation (RAG) application,
the Generative
,AI Engineer needs a structured approach to evaluating the chunking strategy, ensuring that
the chosen configuration retrieves the most relevant information and leads to accurate and
coherent LLM responses. Here's why C and E are the correct strategies:
Strategy C: Evaluation Metrics (Recall, NDCG)
Define an evaluation metric: Common evaluation metrics such as recall, precision, or NDCG
(Normalized Discounted Cumulative Gain) measure how well the retrieved chunks match
the user's query and the expected response.
Recall measures the proportion of relevant information retrieved.
NDCG is often used when you want to account for both the relevance of retrieved chunks
and the ranking or order in which they are retrieved.
Experiment with chunking strategies: Adjusting chunking strategies based on text structure
(e.g., splitting by paragraph, chapter, or a fixed number of tokens) allows the engineer to
experiment with various ways of slicing the text. Some chunks may better align with the
user's query than others. Evaluate performance: By using recall or NDCG, the engineer can
methodically test various chunking strategies to identify which one yields the highest
performance. This ensures that the chunking method provides the most relevant information
when embedding and retrieving data from the vector store.
Strategy E: LLM-as-a-Judge Metric
Use the LLM as an evaluator: After retrieving chunks, the LLM can be used to evaluate the
quality of answers based on the chunks provided. This could be framed as a "judge"
function, where the LLM compares how well a given chunk answers previous user queries.
Optimize based on the LLM's judgment: By having the LLM assess previous answers and
rate their relevance and accuracy, the engineer can collect feedback on how well different
chunking configurations perform in real-world scenarios.
This metric could be a qualitative judgment on how closely the retrieved information
matches the user's intent.
Tune chunking parameters: Based on the LLM's judgment, the engineer can adjust the
chunk size or structure to better align with the LLM's responses, optimizing retrieval for
future queries.
By combining these two approaches, the engineer ensures that the chunking strategy is
systematically evaluated using both quantitative (recall/NDCG) and qualitative (LLM
judgment) methods. This balanced optimization process results in improved retrieval
relevance and, consequently, better response generation by the LLM.
Question: 2
A Generative AI Engineer is designing a RAG application for answering user questions on
technical regulations as they learn a new sport.
What are the steps needed to build this RAG application and deploy it?
A. Ingest documents from a source –> Index the documents and saves to Vector
Search –> User submits queries against an LLM –> LLM retrieves relevant documents –>
Evaluate model –> LLM generates a response –> Deploy it using Model Serving
,B. Ingest documents from a source –> Index the documents and save to Vector Search
–> User submits queries against an LLM –> LLM retrieves relevant documents –> LLM
generates a response -> Evaluate model –> Deploy it using Model Serving
C. Ingest documents from a source –> Index the documents and save to Vector Search
–> Evaluate model
–> Deploy it using Model Serving
D. User submits queries against an LLM –> Ingest documents from a source –> Index
the documents and save to Vector Search –> LLM retrieves relevant documents –> LLM
generates a response –> Evaluate model –> Deploy it using Model Serving
Answer: B
Explanation:
The Generative AI Engineer needs to follow a methodical pipeline to build and deploy a
Retrieval- Augmented Generation (RAG) application. The steps outlined in option B
accurately reflect this process: Ingest documents from a source: This is the first step,
where the engineer collects documents (e.g., technical regulations) that will be used for
retrieval when the application answers user questions. Index the documents and save to
Vector Search: Once the documents are ingested, they need to be embedded using a
technique like embeddings (e.g., with a pre-trained model like BERT) and stored in a vector
database (such as Pinecone or FAISS). This enables fast retrieval based on user queries.
User submits queries against an LLM: Users interact with the application by submitting their
queries.
These queries will be passed to the LLM.
LLM retrieves relevant documents: The LLM works with the vector store to retrieve the most
relevant documents based on their vector representations.
LLM generates a response: Using the retrieved documents, the LLM generates a response
that is tailored to the user's question.
Evaluate model: After generating responses, the system must be evaluated to ensure the
retrieved documents are relevant and the generated response is accurate. Metrics such as
accuracy, relevance, and user satisfaction can be used for evaluation.
Deploy it using Model Serving: Once the RAG pipeline is ready and evaluated, it is deployed
using a model-serving platform such as Databricks Model Serving. This enables real-time
inference and response generation for users.
By following these steps, the Generative AI Engineer ensures that the RAG application is
both efficient and effective for the task of answering technical regulation questions.
Question: 3
A Generative AI Engineer just deployed an LLM application at a digital marketing company
that assists with answering customer service inquiries.
Which metric should they monitor for their customer service LLM application in production?
, A. Number of customer inquiries processed per unit of time
B. Energy usage per query
C. Final perplexity scores for the training of the model
D. HuggingFace Leaderboard values for the base LLM
Answer: A
Explanation:
When deploying an LLM application for customer service inquiries, the primary focus is on
measuring the operational efficiency and quality of the responses. Here's why A is the
correct metric: Number of customer inquiries processed per unit of time: This metric tracks
the throughput of the customer service system, reflecting how many customer inquiries the
LLM application can handle in a given time period (e.g., per minute or hour). High
throughput is crucial in customer service applications where quick response times are
essential to user satisfaction and business efficiency.
Real-time performance monitoring: Monitoring the number of queries processed is an
important part of ensuring that the model is performing well under load, especially during
peak traffic times. It also helps ensure the system scales properly to meet demand.
Why other options are not ideal:
B . Energy usage per query: While energy efficiency is a consideration, it is not the primary
concern for a customer-facing application where user experience (i.e., fast and accurate
responses) is critical. C . Final perplexity scores for the training of the model: Perplexity is a
metric for model training, but it doesn't reflect the real-time operational performance of an
LLM in production.
D . HuggingFace Leaderboard values for the base LLM: The HuggingFace Leaderboard is
more relevant during model selection and benchmarking. However, it is not a direct
measure of the model's performance in a specific customer service application in
production.
Focusing on throughput (inquiries processed per unit time) ensures that the LLM application
is meeting business needs for fast and efficient customer service responses.
Question: 4
A Generative AI Engineer is building a Generative AI system that suggests the best
matched employee team member to newly scoped projects. The team member is selected
from a very large team. The match should be based upon project date availability and how
well their employee profile matches the project scope. Both the employee profile and
project scope are unstructured text. How should the Generative Al Engineer architect their
system?
2026-2028 Latest Version: 6.0 questions with verified answers
and rationales | instant pdf download
Question: 1
A Generative Al Engineer has created a RAG application to look up answers to questions
about a series of fantasy novels that are being asked on the author’s web forum. The
fantasy novel texts are chunked and embedded into a vector store with metadata (page
number, chapter number, book title), retrieved with the user’s query, and provided to an
LLM for response generation. The Generative AI Engineer used their intuition to pick the
chunking strategy and associated configurations but now wants to more methodically
choose the best values.
Which TWO strategies should the Generative AI Engineer take to optimize their chunking
strategy and parameters? (Choose two.)
A. Change embedding models and compare performance.
B. Add a classifier for user queries that predicts which book will best contain the answer.
Use this to filter retrieval.
C. Choose an appropriate evaluation metric (such as recall or NDCG) and experiment
with changes in the chunking strategy, such as splitting chunks by paragraphs or
chapters.
Choose the strategy that gives the best performance metric.
D. Pass known questions and best answers to an LLM and instruct the LLM to provide the
best token count. Use a summary statistic (mean, median, etc.) of the best token counts
to choose chunk size.
E. Create an LLM-as-a-judge metric to evaluate how well previous questions are
answered by the most appropriate chunk. Optimize the chunking parameters based
upon the values of the metric.
Answer: C, E
Explanation:
To optimize a chunking strategy for a Retrieval-Augmented Generation (RAG) application,
the Generative
,AI Engineer needs a structured approach to evaluating the chunking strategy, ensuring that
the chosen configuration retrieves the most relevant information and leads to accurate and
coherent LLM responses. Here's why C and E are the correct strategies:
Strategy C: Evaluation Metrics (Recall, NDCG)
Define an evaluation metric: Common evaluation metrics such as recall, precision, or NDCG
(Normalized Discounted Cumulative Gain) measure how well the retrieved chunks match
the user's query and the expected response.
Recall measures the proportion of relevant information retrieved.
NDCG is often used when you want to account for both the relevance of retrieved chunks
and the ranking or order in which they are retrieved.
Experiment with chunking strategies: Adjusting chunking strategies based on text structure
(e.g., splitting by paragraph, chapter, or a fixed number of tokens) allows the engineer to
experiment with various ways of slicing the text. Some chunks may better align with the
user's query than others. Evaluate performance: By using recall or NDCG, the engineer can
methodically test various chunking strategies to identify which one yields the highest
performance. This ensures that the chunking method provides the most relevant information
when embedding and retrieving data from the vector store.
Strategy E: LLM-as-a-Judge Metric
Use the LLM as an evaluator: After retrieving chunks, the LLM can be used to evaluate the
quality of answers based on the chunks provided. This could be framed as a "judge"
function, where the LLM compares how well a given chunk answers previous user queries.
Optimize based on the LLM's judgment: By having the LLM assess previous answers and
rate their relevance and accuracy, the engineer can collect feedback on how well different
chunking configurations perform in real-world scenarios.
This metric could be a qualitative judgment on how closely the retrieved information
matches the user's intent.
Tune chunking parameters: Based on the LLM's judgment, the engineer can adjust the
chunk size or structure to better align with the LLM's responses, optimizing retrieval for
future queries.
By combining these two approaches, the engineer ensures that the chunking strategy is
systematically evaluated using both quantitative (recall/NDCG) and qualitative (LLM
judgment) methods. This balanced optimization process results in improved retrieval
relevance and, consequently, better response generation by the LLM.
Question: 2
A Generative AI Engineer is designing a RAG application for answering user questions on
technical regulations as they learn a new sport.
What are the steps needed to build this RAG application and deploy it?
A. Ingest documents from a source –> Index the documents and saves to Vector
Search –> User submits queries against an LLM –> LLM retrieves relevant documents –>
Evaluate model –> LLM generates a response –> Deploy it using Model Serving
,B. Ingest documents from a source –> Index the documents and save to Vector Search
–> User submits queries against an LLM –> LLM retrieves relevant documents –> LLM
generates a response -> Evaluate model –> Deploy it using Model Serving
C. Ingest documents from a source –> Index the documents and save to Vector Search
–> Evaluate model
–> Deploy it using Model Serving
D. User submits queries against an LLM –> Ingest documents from a source –> Index
the documents and save to Vector Search –> LLM retrieves relevant documents –> LLM
generates a response –> Evaluate model –> Deploy it using Model Serving
Answer: B
Explanation:
The Generative AI Engineer needs to follow a methodical pipeline to build and deploy a
Retrieval- Augmented Generation (RAG) application. The steps outlined in option B
accurately reflect this process: Ingest documents from a source: This is the first step,
where the engineer collects documents (e.g., technical regulations) that will be used for
retrieval when the application answers user questions. Index the documents and save to
Vector Search: Once the documents are ingested, they need to be embedded using a
technique like embeddings (e.g., with a pre-trained model like BERT) and stored in a vector
database (such as Pinecone or FAISS). This enables fast retrieval based on user queries.
User submits queries against an LLM: Users interact with the application by submitting their
queries.
These queries will be passed to the LLM.
LLM retrieves relevant documents: The LLM works with the vector store to retrieve the most
relevant documents based on their vector representations.
LLM generates a response: Using the retrieved documents, the LLM generates a response
that is tailored to the user's question.
Evaluate model: After generating responses, the system must be evaluated to ensure the
retrieved documents are relevant and the generated response is accurate. Metrics such as
accuracy, relevance, and user satisfaction can be used for evaluation.
Deploy it using Model Serving: Once the RAG pipeline is ready and evaluated, it is deployed
using a model-serving platform such as Databricks Model Serving. This enables real-time
inference and response generation for users.
By following these steps, the Generative AI Engineer ensures that the RAG application is
both efficient and effective for the task of answering technical regulation questions.
Question: 3
A Generative AI Engineer just deployed an LLM application at a digital marketing company
that assists with answering customer service inquiries.
Which metric should they monitor for their customer service LLM application in production?
, A. Number of customer inquiries processed per unit of time
B. Energy usage per query
C. Final perplexity scores for the training of the model
D. HuggingFace Leaderboard values for the base LLM
Answer: A
Explanation:
When deploying an LLM application for customer service inquiries, the primary focus is on
measuring the operational efficiency and quality of the responses. Here's why A is the
correct metric: Number of customer inquiries processed per unit of time: This metric tracks
the throughput of the customer service system, reflecting how many customer inquiries the
LLM application can handle in a given time period (e.g., per minute or hour). High
throughput is crucial in customer service applications where quick response times are
essential to user satisfaction and business efficiency.
Real-time performance monitoring: Monitoring the number of queries processed is an
important part of ensuring that the model is performing well under load, especially during
peak traffic times. It also helps ensure the system scales properly to meet demand.
Why other options are not ideal:
B . Energy usage per query: While energy efficiency is a consideration, it is not the primary
concern for a customer-facing application where user experience (i.e., fast and accurate
responses) is critical. C . Final perplexity scores for the training of the model: Perplexity is a
metric for model training, but it doesn't reflect the real-time operational performance of an
LLM in production.
D . HuggingFace Leaderboard values for the base LLM: The HuggingFace Leaderboard is
more relevant during model selection and benchmarking. However, it is not a direct
measure of the model's performance in a specific customer service application in
production.
Focusing on throughput (inquiries processed per unit time) ensures that the LLM application
is meeting business needs for fast and efficient customer service responses.
Question: 4
A Generative AI Engineer is building a Generative AI system that suggests the best
matched employee team member to newly scoped projects. The team member is selected
from a very large team. The match should be based upon project date availability and how
well their employee profile matches the project scope. Both the employee profile and
project scope are unstructured text. How should the Generative Al Engineer architect their
system?
