Reliable LLM evaluation is less about finding one perfect score and more about building a repeatable decision system. This guide shows how to assemble task-specific test sets, choose useful metrics, estimate quality, latency, and cost, review failures, and set regression checks before an AI application reaches production.
Overview
An LLM application can produce fluent answers while still failing the task that matters. A support assistant may sound helpful but invent policy details; a coding tool may generate valid syntax that introduces a security flaw; a retrieval-augmented generation (RAG) system may cite a relevant document but answer a different question. Evaluation should therefore measure the complete application, not just the underlying model.
A practical LLM evaluation framework has five layers:
- Task success: Did the application provide the required answer, classification, transformation, or action?
- Grounding and safety: Was the response supported by permitted information and free from unacceptable behaviour?
- Consistency: Does the application behave acceptably across representative inputs and repeated runs?
- Operational performance: How long did it take, how many tokens or other billable units did it use, and how often did it fail?
- Regression control: Did a prompt, model, retrieval, tool, or code change make an important case worse?
There is no universal passing score. A useful threshold depends on the task, risk, user expectations, and the cost of an error. A low-risk drafting assistant may tolerate occasional manual correction, while an application that extracts financial or operational data may require strict validation and human approval.
For prompt changes, combine this framework with the LLM evaluation guide and keep structured-output requirements explicit. When an application returns JSON, SQL, or CSV, the structured-output prompt templates provide useful patterns for validating format separately from content quality.
How to estimate
1. Define the unit of success
Write down what one successful test case means before selecting a metric. For example:
- A classifier assigns the correct category and confidence band.
- A summariser includes the required facts without adding unsupported claims.
- A RAG answer responds to the question using evidence from the approved documents.
- A coding assistant produces a solution that passes tests and does not introduce a known security issue.
- An extraction workflow returns valid fields that match the source record.
Separate hard requirements from preferences. “Must return valid JSON” is a hard requirement; “should be concise” is usually a graded preference. This distinction prevents a polished but invalid response from receiving an overly generous score.
2. Build a representative test set
Start with real, anonymised examples where possible. Add deliberately difficult cases rather than relying only on common inputs. A balanced set may include:
- Typical requests that represent normal traffic.
- Boundary cases, such as empty fields, long inputs, ambiguous wording, and unusual formats.
- Known production failures and their corrected versions.
- Adversarial or unsafe requests relevant to the application.
- Cases that test refusal, uncertainty, citations, tool use, or escalation.
Keep a stable core set for prompt regression testing. You can maintain a separate growth set for newly discovered failures so that the core remains comparable between releases. Record the source, expected behaviour, risk category, and whether the expected answer is exact or allows variation.
3. Match metrics to the task
| Evaluation target | Useful measures | Important limitation |
|---|---|---|
| Exact fields or labels | Exact match, accuracy, precision, recall, F1 | Small formatting differences may hide a correct answer, or a correct label may conceal an unsafe explanation. |
| Free-text answers | Rubric score, pairwise preference, semantic similarity | Similarity is not proof of factual correctness or completeness. |
| Summaries | Coverage, factuality, omission and unsupported-claim checks | A short summary can score well while leaving out a critical detail. |
| RAG applications | Retrieval recall, citation validity, groundedness, answer relevance | Good retrieval cannot compensate for an answer that misuses the evidence. |
| Production operation | Latency percentiles, error rate, token usage, cost per task | An average can hide slow or expensive outliers. |
Use automated metrics for scale and human review for ambiguity, nuance, and high-risk failures. An LLM judge can help classify large volumes of responses, but its rubric, model, and disagreement rate should be tested. It should not be treated as an unquestionable ground truth.
4. Calculate a release decision
A simple evaluation report can include:
- Quality rate: passing cases divided by evaluated cases.
- Critical failure rate: critical failures divided by evaluated cases.
- Cost per task: total model and supporting-service cost divided by completed tasks.
- Failure-adjusted cost: operational cost plus the estimated handling cost of failed cases.
- Change impact: new score minus the score from the current production version.
Do not collapse every measure into one number unless the weighting is explicit. A weighted score can be useful for comparing candidates, but a critical safety or formatting failure should usually act as a gate rather than being averaged away.
Inputs and assumptions
Document the conditions under which the evaluation was run. At minimum, capture:
- Model identifier, version or deployment reference, and configuration settings.
- System prompt, user prompt template, examples, tools, and output schema.
- Retrieval index or document snapshot, chunking settings, and search parameters.
- Test-set version, case count, category distribution, and expected outputs.
- Sampling settings, retry rules, timeout values, and fallback behaviour.
- Latency measurement method, including whether queue and network time are included.
- Input and output usage assumptions used for cost calculations.
- Human-review rubric, reviewer instructions, and adjudication process.
Pricing and usage inputs should be variables, not permanent constants. If your provider charges different rates for input and output usage, calculate them separately:
estimated cost = (input units × input rate) + (output units × output rate) + supporting-service costs
For a monthly estimate, add expected volume and operational overhead:
monthly cost = tasks per month × cost per task + fixed platform costs + review or escalation costs
These are planning formulas, not guarantees. Actual bills may vary because of retries, caching, minimum charges, rounding, tool calls, batch processing, or changes to provider pricing. Record the date and source of every pricing input, then recalculate when those inputs change.
A reusable test-case record might look like this:
{
"case_id": "support-014",
"category": "policy_boundary",
"input": "User question or structured payload",
"expected_behaviour": "Answer only from approved policy and state uncertainty",
"reference_answer": "Optional answer or required facts",
"must_pass": ["no unsupported policy claim", "valid JSON"],
"risk": "high",
"metrics": ["groundedness", "schema_validity", "latency"],
"notes": "Added after a production failure"
}For security-sensitive workflows, include prompt-injection and data-exfiltration cases. The prompt injection testing guide can help turn likely attack patterns into repeatable checks.
Worked examples
Example 1: Structured extraction
Suppose an application extracts fields from incoming service requests. The test set contains 200 cases, including 40 boundary or malformed inputs. The release criteria might be:
- Schema validity on every accepted response.
- At least 95% field-level agreement on required fields.
- No critical misclassification in the high-risk category.
- Median latency within the product team's agreed limit.
Run the current version and candidate version on the same test-set snapshot. If the candidate improves field-level agreement but introduces two critical misclassifications, it fails the release gate. Investigate those cases before considering a broader rollout. A schema validator can identify formatting failures, but human or rule-based review is still needed to check whether the extracted value is correct.
Example 2: RAG question answering
For a documentation assistant, score each case independently for retrieval, evidence use, and answer quality. A response may retrieve the correct page but cite an irrelevant paragraph. Another may give a plausible answer without any supporting evidence. Track these as different failure types because they require different fixes: search configuration, chunking, prompt instructions, or answer-generation logic.
Use a rubric with observable questions: “Does the answer address the user’s question?”, “Are all material claims supported by the supplied context?”, and “Does the response clearly state when the context is insufficient?” The RAG prompting best-practices guide offers additional patterns for grounded, cited answers.
Example 3: Cost and quality trade-off
Compare two candidate configurations using the same cases and workload assumptions. Configuration A may have lower estimated cost per task but require more retries or manual corrections. Configuration B may consume more output units but reduce escalation. Report both direct and failure-adjusted cost rather than choosing solely on the provider rate.
Record quality, critical failures, latency, and estimated cost side by side. This makes the decision auditable and allows the team to revisit it when traffic, model rates, or user expectations change.
When to recalculate
Run the full evaluation before releasing a new model, prompt, retrieval configuration, tool integration, output schema, or fallback path. A smaller smoke test is useful for every code change, but it should not replace the full suite when the application behaviour could change materially.
Recalculate cost and capacity estimates whenever provider pricing, context limits, tokenisation behaviour, traffic volume, retry rates, or model configuration changes. Recheck latency after changes to prompts, retrieval, tools, regions, or concurrency. Refresh the test set whenever production reveals a new failure, users adopt a new workflow, or the application's domain changes.
At a minimum, review the evaluation plan quarterly and after major model or API releases. Do not assume that a new model is automatically better for your task: rerun the same core set, inspect category-level results, and compare critical failures against the current baseline.
Release checklist
- Freeze and version the test-set snapshot.
- Run the candidate and baseline under documented settings.
- Check task quality, critical failures, grounding, safety, latency, and cost.
- Inspect regressions by category, not only the aggregate score.
- Review a sample of automated passes and failures manually.
- Confirm that hard gates and rollback conditions are satisfied.
- Store prompts, configuration, results, and reviewer notes with the release record.
- Add important production failures to the regression set.
A dependable LLM testing framework is a maintained engineering asset. Keep its assumptions visible, treat evaluation results as evidence rather than certainty, and make every model or prompt change prove its value against the same practical standards.
