"It looks better" is not engineering. The architect's discipline is to build an eval — a
set of test cases with a scoring method — before tweaking the prompt, so every change is measured,
not vibed. Run a prompt against a golden set and watch a tweak help… or quietly cause a regression.
golden setgradersLLM-as-judgeregressionmeasure before tweak
Explain like I'm 10
Imagine practising free throws. If you just feel like you're improving, you'll fool yourself. So you
keep score: 10 shots, count the makes. Now when you change how you throw, the score tells you the truth
— better or worse. An eval is keeping score for prompts: a fixed list of test questions you re-grade
every time you change something.
Run the eval — then change the prompt
A 6-case golden set scores a support-summary prompt on whether it captured the issue, the resolution, and
the right sentiment. Try each prompt version and read the score. The "clever" rewrite feels better but
drops a case — exactly the regression an eval catches and your gut doesn't:
score
Anatomy of an eval
Piece
What it is
Golden set
20–100+ representative inputs with expected outputs (or rubric). Include the hard/edge cases.
Grader
How each output is scored: exact-match, code check, or LLM-as-judge for open-ended quality.
Metric & bar
One number you optimize (e.g. % correct) and a target ("ship at ≥ 0.9").
Regression gate
Re-run on every prompt/model change; block changes that lower the score.
Exam trap: "the new prompt looks better, ship it" is always wrong — you
measure first. When outputs are open-ended (summaries, tone), the grader is often LLM-as-judge
with a clear rubric, not exact string match. And a change that raises the average but breaks an edge case is a
regression — which is exactly why you keep the hard cases in the set.
Takeaways: build an eval before tweaking — a golden set (with edge
cases), a grader (exact-match, code, or LLM-as-judge), a metric + bar, and a
regression gate. Measure every prompt or model change; trust the score, not the vibe. A higher average
that breaks a case is still a regression.