Testing & A/B Comparing Prompts

Moving from gut-feel prompt tweaking to systematic evaluation

Most prompt iteration is guesswork

The typical way people improve a prompt is: run it, eyeball the output, tweak a sentence, run it again, decide it’s “better.” This works for obvious problems but breaks down for subtle ones — and it’s especially unreliable because a single run can look great or terrible due to randomness that has nothing to do with the change you made. Systematic testing means comparing prompt variants across multiple cases with defined criteria, not vibes on a single output.

Why a single comparison run misleads you

Claude’s output varies somewhat between identical calls. If you change a prompt and the next single output looks better, you can’t tell whether the change helped or you just got a good roll. The fix is running each variant multiple times, or across multiple different inputs, before drawing a conclusion.

Build a test set before you test variants

Before comparing prompt A to prompt B, assemble 5-10 representative inputs that cover the range of what the prompt will actually face in production — including edge cases, not just the easy typical case.

TEST SET for customer email classification prompt:
1. A clear, simple billing complaint (easy case)
2. An email with two unrelated issues mixed together (edge case)
3. A vague email with no clear ask (edge case)
4. An angry, all-caps complaint (tone edge case)
5. An email in a mix of English and another language (edge case)

A prompt that wins on the easy case but breaks on the edge cases isn’t actually better — and you’ll never find that out testing on one friendly example.

Define your scoring criteria before you look at outputs

Decide what “better” means before running the comparison, or you’ll unconsciously rationalize whichever output you already preferred. Write the criteria down as a short rubric.

SCORING RUBRIC (1-5 each):
- Accuracy: correctly classified?
- Format compliance: matches required JSON schema exactly?
- Conciseness: no unnecessary text in the reasoning field?

Run both prompt variants across all 5 test inputs. Score each
output against the rubric before comparing totals.

Using Claude itself as a judge

For larger-scale comparisons, you can have Claude score outputs against your rubric rather than doing it by hand for every case — this is especially useful once your test set gets past a handful of cases. Give the judge prompt the rubric and both outputs, blinded (don’t label which is variant A vs. B) to avoid biasing the judgment.

Here are two responses to the same task. Score each against this
rubric: [paste rubric]. Don't assume either is better going in —
evaluate independently, then state which scores higher and why.

Response 1: [output A]
Response 2: [output B]

Testing constraint adherence, not just quality

For prompts with hard constraints (Lesson 8), your test set should specifically include inputs likely to stress those constraints — the longest realistic input, the most ambiguous one, the one closest to the word limit boundary — since that’s where variants tend to actually diverge.