LLM Prompt Evaluation: A Practical Framework, Scorecard, and Testing Workflow

Overview

LLM evaluation is the process of checking whether a model and prompt produce useful, accurate, safe, and consistent results for a defined task. Prompt evaluation applies the same principle specifically to the instructions, examples, context, and output requirements that shape a model's response.

Casual testing often involves sending a few example requests and choosing the response that feels best. That can help during early exploration, but it makes prompt changes difficult to judge. A response may improve in one case while becoming less accurate, less complete, or less consistent in another. A repeatable evaluation workflow turns that subjective comparison into a documented development process.

The basic workflow is:

1. Define what a successful response must achieve.
2. Create a test set that represents real inputs and known edge cases.
3. Record the prompt, model settings, context, and expected behaviour.
4. Score each output against explicit criteria.
5. Compare prompt versions and investigate regressions.
6. Keep the evaluation set current as the application changes.

For a broader introduction to quality, accuracy, and reliability checks, see the LLM evaluation guide. This article focuses on a reusable scorecard and testing workflow that can be adapted to a developer tool, support assistant, extraction pipeline, or other AI application.

Template structure

Keep each evaluation case small enough to understand and structured enough to compare. A spreadsheet, JSON file, database table, or test fixture can work. The important point is that the same fields are recorded for every test.

1. Test case definition

{
  "case_id": "support-001",
  "category": "routine_request",
  "input": "Can I change the email address on my account?",
  "context": "Approved account-help documentation",
  "expected_behaviour": [
    "Answers the account-change question",
    "Uses only supplied documentation",
    "Explains the next step clearly"
  ],
  "must_not": [
    "Invent a policy",
    "Request unnecessary sensitive information",
    "Claim that an action was completed"
  ]
}

Use categories that reflect how the feature is actually used. For example, an extraction workflow might include complete records, incomplete records, conflicting fields, long documents, and irrelevant text. A coding assistant might include valid requests, ambiguous requirements, unsafe operations, and inputs containing malformed code.

2. Prompt and run metadata

Record the prompt version alongside each result. Include the system instructions, user input, retrieved context if applicable, model identifier, relevant settings, tool calls, and the date of the run. Without this information, a failed result may be impossible to reproduce.

A version label such as extractor-v3 is useful, but a short change note is even better: “Added explicit null handling and prohibited unsupported values.” If you use prompt versioning in a team, the prompt versioning best practices guide provides a useful companion workflow.

3. Scorecard

Score only criteria that matter to the task. A practical five-part scorecard might use a zero-to-two scale:

• Correctness: Does the response reach the right conclusion or produce the right fields?
• Completeness: Does it address all required parts of the request?
• Grounding: Is it supported by the supplied context rather than invented?
• Format compliance: Does it follow the required schema, length, and style?
• Safety and handling: Does it avoid prohibited actions and handle uncertainty appropriately?

Define the scale before testing. For example, zero means a clear failure, one means partially acceptable or requiring review, and two means it meets the criterion. Add a notes field for the reason behind each score. A total score is useful for sorting results, but the individual scores reveal what actually changed.

How to customize

Start with the application's failure modes rather than generic quality categories. Ask what would cause a user, downstream service, or reviewer to reject the result. Those failure modes should become test categories and scorecard criteria.

Build a representative test set

Include ordinary cases, difficult cases, and cases that previously failed. A small but varied set is more useful than a large collection of nearly identical examples. Separate the set into at least three groups:

• Core cases: Common inputs that represent the main workflow.
• Boundary cases: Empty fields, unusual wording, long inputs, ambiguous requests, and missing context.
• Adversarial or misuse cases: Conflicting instructions, unsupported requests, prompt injection attempts, or content that should be refused or escalated.

Keep a portion of the test set stable so that prompt versions can be compared over time. You can maintain another portion for exploration, but avoid changing every test at once; otherwise, it becomes difficult to tell whether the prompt or the dataset caused a difference. The guide on building a prompt evaluation dataset covers this dataset-design problem in more detail.

Choose the right evaluation method

Use deterministic checks where possible. JSON parsing, required-field checks, allowed-value checks, string matching, and schema validation can identify clear failures without another model's judgement. A JSON formatter and validator can help inspect structured results during development, although automated validation should be part of the application or test runner.

Use human review for criteria that require context, such as helpfulness, reasoning quality, tone, or whether an answer is sufficiently cautious. Model-based grading can assist with larger test sets, but it should use a fixed rubric and periodic human checks. Treat a grader's score as evidence to inspect, not as an unquestionable ground truth.

Set release thresholds

Decide in advance what counts as acceptable. For instance, a release might require every structured-output case to pass schema validation, no critical safety case to fail, and the average score for correctness to remain at or above the previous version. The exact thresholds depend on the task and its consequences.

Do not hide serious failures inside an average. A prompt with a high overall score may still produce one unacceptable result in a high-risk category. Track critical failures separately and make them visible in the evaluation report.

Examples

Example: structured information extraction

Suppose a prompt extracts order details from support emails. The scorecard could check whether the order identifier, issue type, urgency, and requested action are present; whether values are copied from the email; and whether unknown fields are returned as null rather than guessed.

A regression test should include an email where the customer mentions two order numbers. The expected result might require the model to flag the ambiguity instead of selecting one silently. It should also include an email with no order number, so the test verifies that the prompt handles missing information consistently. For related extraction patterns, see prompts for extracting information from documents and support tickets.

Example: SQL generation

For a natural-language-to-SQL prompt, evaluate more than whether the query runs. Check that the query uses the permitted tables, applies the requested filters, avoids destructive statements, and returns the requested fields. Test ambiguous date ranges, missing table information, and requests that should result in a clarification rather than a guessed query.

Store the generated SQL separately from any execution result. A query can be syntactically valid but logically wrong. Formatting the query with an SQL formatter may make manual review easier, but it does not replace semantic checks or execution in a controlled environment. The SQL prompt engineering checklist offers additional criteria for this use case.

Example: retrieval-augmented responses

For a RAG workflow, score retrieval and generation separately where practical. Check whether the relevant source passage was retrieved, whether the answer is supported by that passage, and whether the response states when the context does not contain an answer. This separation helps identify whether a failure belongs to search, context assembly, or the prompt itself.

When to update

Run the evaluation suite whenever you change the prompt, model, model settings, retrieval logic, tool definitions, output schema, or surrounding application code. A change to input preprocessing or context selection can alter output quality even when the prompt text remains unchanged.

Update the test set when users introduce a new request type, production monitoring reveals a failure, the product adds a workflow, or the acceptable output changes. Add the failed real-world case, remove or revise cases that no longer represent the product, and document why the change was made. Avoid editing an expected answer merely to make a new prompt pass; first confirm that the expected behaviour is still correct.

Make prompt regression testing part of the normal development workflow. A practical sequence is to run fast deterministic checks on every change, run the full representative set before release, review any critical failures manually, and retain the results for comparison with the previous version. Revisit the rubric when the task's risks or user expectations change.

To put this into practice, create a first set of 15 to 30 varied cases, write a short definition for each score, and record a baseline using the current prompt. Then change one factor at a time and compare the results. This gives your team a clear starting point for improving AI output quality without relying on memory or isolated examples.