ACCELERATED REVIEWS E077 / FREE PREVIEW
Introduction to AI Engineering: Preview Lesson
This independent lesson shows how an AI engineer evaluates a useful decision, not just a model score. It is a
complete sample from an original study companion about applied AI systems. It is not affiliated with WGU, and it
does not claim to match every assessment topic.
Lesson: A score is only part of the decision
Imagine a community center receives maintenance requests from staff during staffed hours. The center wants to
flag reports that need same-day attention. The input is the text available when the request arrives; the action is a
supervisor review within 30 minutes of each arrival. A simple keyword rule is a useful baseline. A classifier may
improve triage, but it also needs valid input, a response-time target, a fallback when the service fails, and a way
for the supervisor to override its flag.
To evaluate the classifier, first define positive as "truly needs same-day attention." Every reviewed request then
belongs to one of four outcomes:
Outcome Meaning
True positive (TP) Urgent request correctly flagged
False positive (FP) Routine request incorrectly flagged
False negative (FN) Urgent request missed
True negative (TN) Routine request correctly left unflagged
Suppose reviewers examine 100 requests. The classifier has TP=12, FP=8, FN=3, and TN=77. The four counts
add to 100. Accuracy is the fraction of all requests classified correctly: (12+77)/100 = 89%. Precision asks how
trustworthy a flag is: 12/(12+8) = 60%. Recall asks how many truly urgent cases were found: 12/(12+3) = 80%. F1
balances precision and recall: 212/(212+8+3) = 24/35, about 68.6%.
Which score matters most depends on the cost of mistakes. If a missed urgent report could close the building and
an extra supervisor review takes a minute, recall deserves close attention. If every false alarm sends an expensive
outside contractor, precision and the number of false alarms matter more. The team should compare the model
with its keyword baseline and choose a flagging threshold using a validation set. It should report the counts along
with the percentages, because 80% recall here still means three urgent reports were missed.
The evaluation must resemble real use. A field recorded after the maintenance visit would leak future information
into a prediction made at request time. Repeated reports about the same broken unit should not be allowed to
make the test set artificially easy. Once a design is chosen, reserve a held-out test set for a final estimate. After
deployment, monitor missing fields, response times, and supervisor corrections. A good offline result is evidence
to start a controlled rollout, not a promise that the service will remain correct.
Edition 1.0 - October 2026 1
Introduction to AI Engineering: Preview Lesson
This independent lesson shows how an AI engineer evaluates a useful decision, not just a model score. It is a
complete sample from an original study companion about applied AI systems. It is not affiliated with WGU, and it
does not claim to match every assessment topic.
Lesson: A score is only part of the decision
Imagine a community center receives maintenance requests from staff during staffed hours. The center wants to
flag reports that need same-day attention. The input is the text available when the request arrives; the action is a
supervisor review within 30 minutes of each arrival. A simple keyword rule is a useful baseline. A classifier may
improve triage, but it also needs valid input, a response-time target, a fallback when the service fails, and a way
for the supervisor to override its flag.
To evaluate the classifier, first define positive as "truly needs same-day attention." Every reviewed request then
belongs to one of four outcomes:
Outcome Meaning
True positive (TP) Urgent request correctly flagged
False positive (FP) Routine request incorrectly flagged
False negative (FN) Urgent request missed
True negative (TN) Routine request correctly left unflagged
Suppose reviewers examine 100 requests. The classifier has TP=12, FP=8, FN=3, and TN=77. The four counts
add to 100. Accuracy is the fraction of all requests classified correctly: (12+77)/100 = 89%. Precision asks how
trustworthy a flag is: 12/(12+8) = 60%. Recall asks how many truly urgent cases were found: 12/(12+3) = 80%. F1
balances precision and recall: 212/(212+8+3) = 24/35, about 68.6%.
Which score matters most depends on the cost of mistakes. If a missed urgent report could close the building and
an extra supervisor review takes a minute, recall deserves close attention. If every false alarm sends an expensive
outside contractor, precision and the number of false alarms matter more. The team should compare the model
with its keyword baseline and choose a flagging threshold using a validation set. It should report the counts along
with the percentages, because 80% recall here still means three urgent reports were missed.
The evaluation must resemble real use. A field recorded after the maintenance visit would leak future information
into a prediction made at request time. Repeated reports about the same broken unit should not be allowed to
make the test set artificially easy. Once a design is chosen, reserve a held-out test set for a final estimate. After
deployment, monitor missing fields, response times, and supervisor corrections. A good offline result is evidence
to start a controlled rollout, not a promise that the service will remain correct.
Edition 1.0 - October 2026 1
