Testing & A/B Comparing Prompts — Quick Check

Quiz Summary

0 of 3 Questions completed

Questions:

You have already completed the quiz before. Hence you can not start it again.

Quiz is loading…

You must sign in or sign up to start the quiz.

You must first complete the following:

Results

Quiz complete. Results are being recorded.

Results

0 of 3 Questions answered correctly

Your time:

You have reached 0 of 0 point(s), (0)

Question 1 of 3

1. Question

You tweak a prompt, run it once, and the output looks noticeably better than before. You conclude the change worked. What’s the risk in this conclusion?

There is no risk, one comparison is always sufficient
Correct Incorrect Correct answer

A single run can look better or worse due to normal output variation unrelated to your change — you need multiple runs or varied inputs to know if the change actually helped
Correct Incorrect Correct answer

The risk only applies to creative writing tasks
Correct Incorrect Correct answer

Prompt changes always have a consistent, predictable effect
Correct Incorrect Correct answer

Correct

Incorrect
Question 2 of 3

2. Question

A new prompt variant scores better than the old one on your one go-to test example. Before adopting it, what should you do?

Adopt it immediately, one clear win is enough evidence
Correct Incorrect Correct answer

Test it against a small set of varied inputs, including edge cases, since a change that helps on one easy case can hurt on others
Correct Incorrect Correct answer

Never change the prompt again
Correct Incorrect Correct answer

Ask a different AI model to verify
Correct Incorrect Correct answer

Correct

Incorrect
Question 3 of 3

3. Question

When using Claude itself to judge which of two prompt output variants is better, what practice reduces bias in the judgment?

Telling the judge model which variant is the 'new, improved' one
Correct Incorrect Correct answer

Blinding the labels (not revealing which output is variant A vs. B) and giving the judge a clear rubric to score against
Correct Incorrect Correct answer

Only ever showing the judge the variant you expect to win
Correct Incorrect Correct answer

Skipping a rubric and asking for a general impression
Correct Incorrect Correct answer

Correct

Incorrect

Testing & A/B Comparing Prompts — Quick Check

Quiz Summary

Information

Results

Results

Categories

1. Question

2. Question

3. Question