Prompt chaining turns a complex LLM task into a sequence of smaller, testable operations. This guide explains how to design sequential, branching, and review-based chains; estimate token cost and latency; handle failures; and maintain workflows as models, prompts, and requirements change.
Overview
A single prompt is useful when a task is clear and bounded. It becomes harder to control when the request requires several different forms of reasoning, such as extracting data, checking it against a policy, transforming it into a format, and producing a user-facing answer. Prompt chaining separates those responsibilities into multiple LLM calls connected by explicit inputs and outputs.
A basic chain might look like this:
- Extract facts from the source material.
- Validate that required facts are present.
- Apply a decision rule or classification.
- Generate a response from the validated result.
Each step should have one main job, a defined input, an expected output, and a failure path. This makes the workflow easier to debug than a large prompt that asks the model to perform every operation at once. It also supports more precise LLM prompt evaluation, because each stage can be tested independently.
Prompt chaining is not automatically better than a single prompt. It adds model calls, orchestration logic, validation, and possible latency. Use it when the additional control is more valuable than the simplicity of one request.
How to estimate
Before implementing a chain, estimate three practical outcomes: cost, latency, and usable completion rate. The figures are planning inputs rather than guarantees, so calculate them with the model, provider, traffic pattern, and retry policy you expect to use.
1. Estimate the number of calls
Start with the normal path through the workflow:
Normal calls per item = number of required chain steps
Then add calls caused by retries, reviews, or branches:
Expected calls per item = normal calls + expected retry calls + expected review calls
For a branching workflow, estimate each branch separately and multiply it by the share of items expected to take that route. If 70% of requests take one path and 30% take another, calculate the expected calls for both paths rather than assuming every request uses the longest route.
2. Estimate token use
For each step, record the input tokens and output tokens. Include repeated context, system instructions, retrieved documents, examples, and formatting instructions. A useful planning formula is:
Total tokens per item = sum of input and output tokens across all calls
When a later step receives the full output of an earlier step, count that output again as input for the later call. This is easy to miss in chains that pass large documents or verbose intermediate reasoning.
If a provider exposes separate input and output rates, estimate them independently:
Estimated cost = input tokens × input rate + output tokens × output rate
Apply the relevant unit conversion for the pricing format you are using. Keep the rates in configuration rather than hard-coding them into application logic, so the estimate can be updated without changing the chain.
3. Estimate latency
For sequential calls, a simple upper-level estimate is:
Chain latency = sum of model latency, orchestration time, validation time, and external-tool time for each step
Parallel branches can reduce elapsed time. If two independent steps run concurrently, their contribution is closer to the slower branch rather than the sum of both branches. Add queueing, network, and retry time when planning for production traffic.
4. Estimate usable outcomes
Track whether an item reaches a valid final result, not just whether every model call returns a response. A chain can produce technically valid text that fails a schema check, omits a required field, or needs human review.
Usable completion rate = valid final outcomes ÷ total items
Measure this rate by step as well. A low final completion rate may be caused by extraction errors, an overly strict validator, a weak branch condition, or a final prompt that cannot use the intermediate data.
Inputs and assumptions
A reliable estimate begins with a small worksheet. Capture the following inputs for each chain step:
- Purpose: the single decision or transformation the step performs.
- Input size: typical and worst-case tokens, including retrieved context.
- Output size: the expected response and maximum allowed length.
- Format: plain text, structured JSON, a classification label, or another constrained representation.
- Validation: schema checks, allowed-value checks, citations, business rules, or confidence thresholds.
- Failure action: retry, repair, alternate model, fallback response, or human review.
- Branch rate: the expected proportion of requests following each route.
- Context policy: which earlier outputs are passed forward and which are discarded.
Use structured output prompts when one step feeds another. Ask for named fields with clear types and allowed values, then validate the result in application code. A model instruction such as “return valid JSON” is not a substitute for parsing and validation. The guide to reliable JSON prompts provides a useful companion approach for schemas and error handling.
Keep intermediate outputs compact. Instead of passing a full analysis to the next step, pass only the fields that step needs. This reduces token use and limits accidental coupling between prompts. For retrieval-based workflows, define how documents are selected, how source references are preserved, and what happens when evidence is missing. See the RAG prompting best practices guide for grounded, cited patterns.
Finally, distinguish deterministic checks from model judgements. Use ordinary code for JSON parsing, length limits, regular-expression checks, permissions, and arithmetic. Reserve LLM calls for language tasks that require interpretation or generation.
Worked examples
Sequential support-ticket classification
Suppose a workflow processes an incoming support message in four stages:
- Extract: identify the product, issue, urgency, and requested action.
- Check: determine whether the required fields are present and whether the message needs clarification.
- Route: assign a queue and priority from a controlled list.
- Draft: create a concise reply using the extracted facts and routing result.
The normal path uses four calls. If missing fields trigger a clarification call for an estimated 15% of items, the expected call count becomes 4.15 before retries. If a schema failure causes a single repair attempt on 5% of calls, add the expected repair calls separately. This is more useful than budgeting for four calls exactly, because the estimate reflects the actual workflow.
Keep the extraction output separate from the customer-facing draft. The route step should consume the validated fields, while the draft step should receive only the approved facts and queue guidance. If the draft fails validation, retry that step rather than repeating extraction and routing.
Branching document review
A document workflow may first classify a file as a contract, invoice, or general correspondence. Each category then uses a different extraction prompt. The expected cost is calculated as:
Expected cost = classification cost + (contract share × contract path cost) + (invoice share × invoice path cost) + (correspondence share × correspondence path cost)
Add a review step for low-confidence or incomplete results. Set the review condition using observable signals, such as a missing required field, conflicting values, failed citation checks, or an allowed confidence threshold. Avoid asking the model to decide whether it is “correct” without defining what correctness means.
Review-and-repair pattern
A generation step can be followed by a separate reviewer that checks the output against a short rubric. If the reviewer finds a fixable issue, pass the original output and specific defects to a repair step. If the issue concerns missing source evidence, route back to retrieval rather than asking the writer to invent a correction.
For repeatable testing, build a small evaluation dataset containing normal, ambiguous, incomplete, and adversarial examples. Compare each chain version using the same cases. The guides on building an evaluation dataset and testing prompt quality and reliability can help structure this process.
When to recalculate
Recalculate a prompt chain whenever an input that affects cost, quality, or routing changes. At minimum, review the estimate when model pricing inputs change, when a different model or context window is introduced, or when average document length moves materially. Also revisit it after changing system instructions, few-shot examples, retrieval settings, output limits, or validation rules.
Operational data should trigger a review too. Investigate increases in retry frequency, schema failures, fallback usage, latency, user corrections, or human-review volume. A chain that was economical with short inputs may become expensive when retrieved context grows. A branch that was rare during testing may become the dominant route after a product or policy change.
Use this maintenance checklist:
- Version each prompt, schema, routing rule, and model configuration.
- Record token use, latency, retries, validation failures, and final outcome quality by step.
- Run a fixed evaluation set before releasing prompt changes.
- Compare normal-path and worst-path cost, not only the average.
- Review intermediate outputs for unnecessary context and duplicated instructions.
- Retest branch conditions and fallback behaviour with incomplete and ambiguous inputs.
- Update the estimate when pricing inputs, traffic mix, or benchmark results change.
Start with the smallest chain that provides a measurable control advantage. Make every hand-off explicit, validate every machine-readable output, and keep a record of why each step exists. Those prompt engineering best practices make AI workflow automation easier to cost, test, debug, and maintain as requirements evolve.
