How to reduce LLM latency: TTFT, output length, caching, and model selection

October 27, 2026 · 9 min read

A quick first token does not mean a usable answer. Five tips connect end-to-end measurement, shorter output, prompt caching, context budgets, and model routing in a support assistant.

What should you measure when an LLM answer is slow?

Find the slow stage before changing models. In a fictional support-drafting assistant, record request receipt, first display, completion of a validated evidence-backed draft, and completion of any requested action separately. TTFT measures time to first token, but align the start point and qualifying token across instrumentation.

Measure	Scope	Common blind spot
First display	Time from the defined entrance through queues, retrieval, and model startup	A quick preamble is not a finished answer
Usable answer	Completed retrieval, generation, and validation	Include schema checks and regeneration
Action completion	Authorization, tools, and confirmation of external results	Distinguish acceptance from confirmed success

For one request, serial routing of 100ms, retrieval of 300ms, generation of 1,200ms, and validation of 400ms total 2,000ms. Independent work may run in parallel; generation requiring retrieved evidence cannot finish before that evidence exists. Do not add stage p95 values and label the sum end-to-end p95; measure complete requests.

Shorten output when decoding dominates

Prefill processes input; decoding generates output. Under an illustrative 0.5-second TTFT and roughly 50 output tokens per second, approximate completion as 0.5 + output tokens / 50. That gives 16.5 seconds for 800 tokens and 4.5 seconds for 200. This approximation simplifies treatment of the first token and is not a product speed guarantee.

Illustrative model-call completion time in seconds, excluding retrieval and validation

800 tokens: 16.5 s
200 tokens: 4.5 s

Keep the conclusion, evidence, and next step, while removing repeated explanation. Lowering only the output cap can truncate prose or JSON, so align instructions and format with the shorter answer. Streaming helps readers start sooner, but later validation cannot reliably recall unsuitable text already displayed.

Prompt caching: separate stable and variable content

For prefix caching, reuse depends on matching content from the start. A changing timestamp or request ID before long shared material can disrupt reuse after it. Where supported, put stable tools, instructions, and shared material first, with per-request input later. Authorize supplied documents on every request; caching does not replace access checks.

Conceptual ordering, not a complete API request

stable tools -> stable policy -> authorized shared references
             -> relevant history -> current question and request metadata

Verify minimum cacheable length, boundaries, TTL, and write pricing for the selected API and model. Record cached input tokens, miss latency, and cost including writes. Reusing input does not eliminate output generation; other changes may matter more for short prompts or infrequent calls.

Budget long history and tool results

For a fictional shared 32,000-token input/output window, reserving 8,000 output tokens and 4,000 for instructions and overhead leaves 20,000 for history, retrieval, and tools. These are not product limits. Bound retrieval and tool output according to the actual input, output, and reasoning-token accounting rules.

When compacting history, check for lost amounts, exceptions, and provenance. Preserve critical facts structurally with access to originals. On overflow, narrow retrieval, split stages, or ask for missing constraints instead of silently truncating. Shorter context that loses necessary evidence is not an improvement.

Account for escalation before routing to a smaller model

An upfront router selects one model; a cascade inspects a lighter model’s output before escalating. If every request costs one unit and 20% cost five additional units, the mean is 1 + 0.2 × 5 = 2. These are fictional call costs excluding validation, retrieval, and retries. Escalated requests wait for both stages.

Do not equate model-reported confidence with correctness. Evaluate incorrect acceptance and unnecessary escalation by language, difficulty, and input length. Lower mean cost does not meet adoption criteria if difficult cases get slower or serious errors increase. Distinguish product routing, such as Bedrock’s feature, from this custom cascade.

Validate speed and quality together

Use the same evaluation set for cold calls, cache hits, long conversations, and large tool results.
Link input/output length, queues, retries, and validation attempts to traces; measure end-to-end p50 and p95.
Compare evidence quality, complete structured output, and serious business errors before and after changes.
For work exceeding the overall deadline, compare asynchronous execution with accepted, running, failed, and completed states.

Primary sources

The official sources below were checked on October 27, 2026. Scenarios and numerical examples are illustrative assumptions. Configuration examples are explanatory; they have not been run against live services or benchmarked.