Evaluating the initial output and identifying weaknesses
Evaluate an output against criteria and link each weakness to a likely cause in the prompt.
Look before you revise
The most common iteration mistake is rewriting the prompt at random because the output "isn't quite right". Instead, first describe exactly what is wrong. You cannot fix a problem you have not named.
Evaluate against criteria
Use the success criteria you wrote (Module 4) or the evaluation checklist in your resources. For each criterion, mark met, partly met or not met, and note evidence ("Paragraph 2 recommends a product, which the constraints ban").
Also check for problems you did not anticipate:
- Accuracy — any claim that is false, unsupported or unverifiable?
- Completeness — anything required that is missing?
- Relevance — content you did not ask for?
- Format — structure, length, headings, table columns as specified?
- Tone and audience — right register for the reader?
- Consistency — does it contradict itself or the source?
Diagnose: trace problems to causes
Most weaknesses map to a prompt element:
| Symptom | Likely cause | Typical fix |
|---|---|---|
| Generic, could be about anyone | Missing context | Add purpose, audience, situation |
| Wrong length or shape | Vague format | Structural limits, explicit template |
| Ignores an instruction | Buried, conflicting or too many rules | Move it to the end, resolve conflicts, trim |
| Invented facts or sources | No source material; asked for facts | Provide sources; allow "not stated" |
| Inconsistent across runs | Ambiguous task or labels | Define terms; add examples |
| Copies example wording | Over-specific examples | Vary examples; say "format only" |
| Answers a different question | Ambiguous wording | Name the action and deliverable precisely |
Test more than once
Because outputs vary, one good (or bad) result proves little. For a prompt you will reuse, run it at least three times, ideally on different inputs, including an awkward one (very short input, missing information, an unusual case). Record how many runs met each criterion.
Separate the prompt from the model
Sometimes the problem is not your prompt: the task may exceed what the model can do reliably, the information may not exist in the input, or the tool may lack a feature (such as web access). If several careful revisions do not help, consider changing the task, adding a source, splitting the work or doing that part yourself.
Example evaluation
Goal: a 100-word event description for a library newsletter.
- Criterion "mentions date, time and place" — partly met (no time).
- Criterion "under 100 words" — not met (148 words).
- Unexpected: says "free refreshments", which the brief never mentioned → invented detail.
Diagnosis: the time was not in the source notes (context gap); the length was stated as "short" (vague format); the refreshments were filled in from typical event descriptions (no instruction to use only provided facts).
Practice exercise
Take an output from your log that was not quite right.
Submit:
- Your success criteria and a met / partly met / not met mark for each, with evidence.
- At least two weaknesses traced to likely causes in the prompt (use the table).
- The single change you will try first, and why you chose it over the others.