All posts

How to measure AI product quality: connect acceptance criteria, evals, and production monitoring

October 26, 2026 · 12 min read

A higher average answer score can hide more review work or serious errors. Follow a support AI through agreed outcomes, workflow metrics, LLM evaluations, release gates, configuration versions, and production outcomes, drawing on six related tips.

What is AI product quality? Define when the work is complete

Natural answers can still leave production users spending too much time checking and correcting them. Quality design examines whether people can finish their work, recover from failure, and keep using the product within acceptable time and cost, alongside answer correctness.

Consider a fictional support AI that summarizes inquiries and drafts answers from internal procedures. Staff review and send responses, with specialist handoffs when needed. Completion means meeting the agreed conditions and confirming resolution, including any required work at the receiving team. Producing a draft alone does not complete the task.

Acceptance criteria: specify answers, clarification, deferral, and handoffs

Acceptance criteria define outcomes demonstrating that a user need has been met. GOV.UK’s Service Manual describes them in these terms. For support AI, turn a broad goal such as helpful answers into inputs, permissions, procedure versions, expected behavior, and a grading rationale.

When staff disagree about an expected outcome, check procedures and assumptions with the business owner. A majority label can leave the underlying disagreement unresolved. If no decision is possible, define what the AI should ask and who will take responsibility for the judgment.

Deferral or handoff can pass an evaluation when that is the expected behavior. Count this separately from completed work. Track incorrect guidance, unnecessary deferrals, and final completion together so a system that defers everything does not appear successful.

Quality metrics: measure drafting, review, and correction across the workflow

Use the calculation from the total workflow value tip. A case previously took ten minutes. If people now spend two minutes drafting, five checking, and four correcting, effort is 2 + 5 + 4 = 11 minutes: one minute more at equal scope and quality. Those two minutes measure human drafting effort, not AI inference latency.

GOV.UK describes measuring completion and duration across user journeys. Looking only at successful AI users can hide difficult cases shifted to people. If the population or difficulty changes, a simple before-and-after difference does not establish a causal effect.

LLM evaluation sets: compare changes under the same conditions

An evaluation set pairs inputs with pass criteria. Combine agreed normal, exceptional, and deferred cases with authorized examples and boundary inputs. Tag task, language, permissions, and source version. Separate development and final-check cases. If a final-check failure informs a fix, retain it for regression checks and replenish unseen cases as needed.

Suppose a prompt gains an instruction to be shorter. The LLM evaluation tip describes overall performance rising from 86% to 88% across 300 cases while refunds fall from 92% to 71%. The following illustrative breakdown, added here, reproduces those rates using 100 refund cases and 200 other inquiries.

There are six more passes overall but 21 fewer for refunds. If summaries dropped amounts or order numbers, fix the failure and reevaluate every category. Include category sample sizes and actual failures in release decisions. This is one hypothetical run. To assess whether the two-percentage-point change is stable, repeat the same cases and compare against variation between runs.

When comparing prompts, keep the model, sources, retrieval settings, grading criteria, and other conditions fixed. Bundled configuration changes can also be evaluated on the same cases, but identifying which component caused a difference requires investigation. Do not directly compare aggregates whose case counts or category mix have changed.

Grading: combine code, people, and LLM-as-a-Judge

Anthropic describes combining code, model, and human grading for the task. The original LLM-as-a-Judge paper reports biases including position and verbosity. Specify criteria and grading rationales, then calibrate against domain-reviewed examples. Calibration is needed whether or not the judge shares a model with the evaluated system. Record and evaluate grading-model and prompt changes too.

Quality gates: define release conditions that average gains cannot offset

Quality gates are conditions required before release or configuration changes. Agree unacceptable failures with the business owner, separating mandatory gates from comparative scores. The rules below illustrate this support workflow; they are not universal numerical standards for AI products.

A version failing the refund gate cannot ship for this workflow. Fix missing required information rather than offsetting it with style gains. Conversely, a justified deferral should not fail simply because no answer was supplied. Connect evaluation decisions to completion measurement, including what happens after deferral.

Versioning: connect the evaluated configuration to the deployed one

Even with the same model, changes to prompts, chunking, indexes, retrieval, or tool contracts can alter outcomes. Google Cloud’s operations guide recommends recording prompts, component versions, model versions, and evaluation results. Collect component identifiers in a release manifest alongside the model name.

{
  "release": "support-ai-r12",
  "model": "M2",
  "prompt": "P7",
  "index": "I4",
  "retrieval": "R3",
  "tools": "T2",
  "evaluationSet": "E5",
  "grader": "G3",
  "application": "A8"
}

Link production records to releases and retain references to actual model identity and availability, retrieval assets, and evaluation results. Records support comparison but do not guarantee identical regenerated text. Before rollback, verify tool compatibility and current permissions and deletions. Do not restore an old index unchanged if it would bring back deleted documents.

Production monitoring: follow HTTP success through to business outcomes

An HTTP 200 response saying a case was handed off can coexist with failed intake. OpenTelemetry records work units as spans and correlates operations into traces. Apply this approach to retrieval, generation, validation, external actions, and subsequent business outcomes.

Keep generation success, receiving-team acceptance, and business completion as separate states. Link request IDs, releases, source identifiers, stage durations and results, retries, usage, and external outcomes as needed. Track later completion through a durable case ID. A model’s stated explanation is not evidence that an action succeeded.

OWASP’s Logging Cheat Sheet advises against directly logging access tokens and sensitive information. Define needed information, access controls, and retention rather than assuming full content must be stored. Document and case identifiers can also be sensitive. Design separate, authorized diagnostic collection when content is necessary.

Reconcile new failures with input conditions, configuration versions, and actual outcomes. Check permission to reuse cases and handling of confidential information before adding them to evaluations, then check fixes for regressions elsewhere. Examine repeat inquiries, corrections, and receiving-team burden alongside response ratings.

Getting started: run the quality loop for one workflow

1. Define workflow completion and agree normal, exceptional, and deferred inputs and outcomes with domain staff.
2. Measure current completion, human effort, elapsed time, errors, and cost, including non-use and handoffs.
3. Separate development and final-check cases. Calibrate code, human, and LLM grading; compare changes under the same conditions.
4. Review category counts and failures, apply mandatory gates, and record the evaluated configuration and recovery conditions.
5. Use the system within a limited scope, verify outcomes and workload, and add production failures to subsequent evaluations.

Primary sources on acceptance, evaluation, and operations

• GOV.UK Service Manual — Writing user stories: acceptance criteria
• GOV.UK Service Manual — Measuring the success of your service
• Anthropic — Demystifying evals for AI agents
• Zheng et al. — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Google Cloud — Deploy and operate generative AI applications
• OpenTelemetry — Traces
• OWASP — Logging Cheat Sheet