Testing and Improving Prompts

A prompt that works on one example may fail on the next. Professionals test prompts systematically.

Define success first

Write down what a good output is before you tweak anything:

• Correctness: the facts or classification are right.
• Format: it parses, fits length limits, uses the required fields.
• Tone and safety: on brand, no promises you cannot keep.

Build a test set

Collect 20–50 real inputs, including:

• typical cases,
• edge cases (empty input, very long input, mixed languages),
• tricky cases that broke earlier versions.

Store them in a file with the expected output or grading notes.

Grade outputs

A simple harness:

import json
cases = json.load(open("cases.json"))
passed = 0
for case in cases:
    output = run_prompt(case["input"])        # calls your model
    ok = grade(output, case["expected"])      # your checks
    passed += ok
    if not ok:
        print("FAIL:", case["input"][:60], "->", output[:80])
print(f"{passed}/{len(cases)} passed")

Iterate one change at a time

Change one thing (add an example, clarify a constraint), rerun the whole test set, and keep the change only if the score improves without breaking other cases. Keep prompt versions in Git like code.

Cost and speed

• Shorter prompts are not always cheaper overall: a clearer, longer prompt that succeeds first time beats retries.
• Reuse stable prompt prefixes so the API can cache them (Claude & GPT Integration explains prompt caching).
• Use a smaller, faster model or lower reasoning effort for simple, high-volume tasks, and verify quality holds on your test set.

A prompt library

Keep your best prompts as templates with placeholders, documented with what they are for, which model and settings they were tested with, and their test-set score. Teams that do this ship AI features faster and more reliably.

Final project

Pick a real task from your work (support replies, changelog writing, data extraction). Write a prompt, build a 25-case test set with grading, run three improvement iterations, and document the before/after pass rates and what each change did.

Check yourself

• Why test prompts on many inputs instead of one?
• When is model-graded evaluation appropriate?
• Why change only one thing per iteration?