LLM evaluation turns vague impressions such as “this answer looks better” into repeatable evidence. This guide presents a practical way to test prompt quality, accuracy, consistency, safety, latency, and cost, then monitor those measures as prompts, models, and data change.
Overview
A prompt can work well in a demonstration and still fail in production. Users phrase requests differently, source data changes, edge cases appear, and model updates can alter output behaviour. Effective LLM evaluation therefore tests the complete AI workflow rather than judging a handful of impressive examples.
Start by defining what “good” means for the application. A support-answering system may need accurate citations, appropriate escalation, and a clear tone. A code assistant may need syntactically valid output, adherence to repository conventions, and no invented APIs. A classification workflow may prioritise correct labels, stable formatting, and predictable handling of ambiguous inputs.
Evaluation is easiest to manage when it has four connected parts:
- A representative dataset: real or carefully constructed inputs covering normal, difficult, and failure-prone cases.
- Explicit criteria: measurable checks or a scoring rubric for each important quality.
- Repeatable execution: the same prompts, model settings, tools, and test conditions whenever possible.
- A review process: a record of results, failures, decisions, and changes over time.
This approach complements prompt engineering. For examples of workflow design, see prompt chaining patterns and error handling, or review structured JSON prompts with schemas and validation.
What to track
1. Build a useful evaluation dataset
Record each test case with an input, expected behaviour, relevant context, and an evaluation method. Include straightforward examples, common variations, known historical failures, boundary cases, and adversarial or ambiguous requests. For a retrieval-augmented application, preserve the documents or passages that should support the answer so that retrieval and generation can be examined separately.
A compact dataset might use fields such as:
- Case ID: a stable identifier for regression tracking.
- Input: the user request and any required metadata.
- Expected result: an answer, label, required facts, or acceptable behaviour.
- Risk level: low, medium, or high impact if the output fails.
- Evaluation type: exact match, rule-based check, semantic comparison, human review, or a combination.
Do not rely only on synthetic examples. They can help fill coverage gaps, but anonymised production cases and manually selected failures usually reveal more about how people actually use the system. Keep sensitive information out of shared evaluation files and document any transformations applied to the data.
2. Measure more than correctness
Accuracy is important, but it is only one dimension of AI reliability testing. Track the measures that reflect the application’s real risk:
- Task success: whether the output solves the user’s stated problem.
- Factuality and grounding: whether claims are supported by supplied context or trusted sources.
- Instruction adherence: whether the model follows constraints, exclusions, tone, and scope.
- Format validity: whether structured output parses and matches the required schema.
- Safety and refusal behaviour: whether unsuitable requests are handled according to the application’s rules.
- Consistency: whether similar inputs receive appropriately similar treatment.
- Operational measures: latency, token use, tool errors, retries, and failure rates.
Use automated checks where the requirement is objective. For example, parse JSON, validate required fields, check citations against available passages, or confirm that a generated code block contains a required function. Use human review for qualities such as helpfulness, nuanced relevance, and whether an answer creates a misleading impression.
3. Use a scoring rubric
A rubric makes human review more consistent. Define a small scale and describe each score. For example, a factuality score of 0 could mean “unsupported or materially wrong,” 1 could mean “partly supported but with a significant omission,” and 2 could mean “supported and complete for the request.” Add a separate flag for critical failures, because an average score can hide one unacceptable answer.
Store reviewer comments alongside scores. A comment such as “correct conclusion, but source passage does not support the second claim” is more useful for prompt improvement than a numeric result alone.
Cadence and checkpoints
Run a small regression set whenever a prompt, model, retrieval configuration, tool definition, parser, or safety rule changes. Run the broader dataset on a regular monthly or quarterly cadence, depending on how often the system changes and how costly failure is.
A practical checkpoint routine is:
- Before a release: run all critical cases and compare results with the current baseline.
- After deployment: sample outputs, check operational metrics, and confirm that logging and fallbacks work.
- Monthly: review new failure reports, add representative cases, and remove tests that no longer reflect the product.
- Quarterly: reassess the rubric, risk categories, model choice, and whether the dataset still represents real usage.
Keep a changelog with the date, prompt or model version, dataset version, score changes, and release decision. Versioning prevents a team from comparing results produced under different conditions without realising it.
How to interpret changes
Do not treat one aggregate score as the whole evaluation. Break results down by case type, risk level, language, input length, retrieval status, and failure category. A stable average may conceal a decline in high-risk cases, while a lower overall score may reflect the addition of harder but more realistic tests.
When a result changes, isolate the cause before rewriting the prompt. Compare the prompt version, model, temperature or other settings, retrieved context, tool responses, parser behaviour, and test data. Re-run the failing case several times when output variability is expected. A failure that occurs once may need a consistency test; a failure that repeats under the same conditions needs a design change.
Separate prompt regressions from application regressions. If the model receives incomplete context, improving the system prompt may not solve the underlying retrieval problem. For grounded applications, the RAG prompting best practices guide offers a useful way to examine source use and citations. For multi-step systems, evaluate each stage as well as the final response; a later step may conceal an earlier extraction or routing error.
Set release thresholds in advance. For example, require every critical case to pass, prevent a material drop in factuality, and allow a small improvement in one measure only if the effect on latency or cost is understood. Thresholds should reflect the application’s risk rather than an arbitrary universal benchmark.
When to revisit
Revisit this evaluation process whenever the system changes, not only when a user reports a visible failure. Add or rerun tests after a model update, prompt rewrite, new tool, retrieval-index change, schema change, data-source update, or change in the types of requests users submit.
Use a monthly review to ask whether recent failures have become test cases and whether the monitoring data still reflects the most important risks. Use a quarterly review to retire stale cases, refresh representative examples, recalibrate human-review guidance, and confirm that critical categories still have enough coverage.
For the next review, take these actions:
- Export a sample of recent failures and remove sensitive information.
- Classify each failure as a data, retrieval, prompt, model, tool, parsing, or user-experience issue.
- Add the most representative cases to the versioned evaluation dataset.
- Run the regression suite against the current baseline and proposed change.
- Record score changes by risk category, not just the overall average.
- Decide whether to release, revise, or roll back, and document the reason.
Consistent LLM evaluation is less about finding one perfect benchmark than about creating a dependable feedback loop. A maintained dataset, clear rubric, structured outputs, regression tests, human review, and regular checkpoints give developers evidence they can use when prompts or models change.
