When should AI abstain? Count the answers it withholds

A system that answers fewer questions can look more accurate. To understand the result, count what it answered, what it withheld and what reviewers could not judge. An error percentage alone leaves the work unfinished.

What does abstention mean?

The survey Know Your Limits describes abstention as an LLM declining to provide an answer. Its opening distinguishes problems with the query, insufficient response confidence and response alignment. These are different reasons to stop; an evidence gap and a prohibited request need different handling. Survey, abstract and introduction.

The Abstain-QA abstract describes evaluation using answerable and unanswerable questions and an Answerable-Unanswerable Confusion Matrix. That supports testing both appropriate withholding and unnecessary withholding. Our reporting convention below is an editorial example, not a reproduction of the paper’s metrics. Research abstract.

When should a business assistant stop or ask for help?

Our proposed rule is to define the required evidence before permitting an answer or action. Use these review outcomes:

• Missing input: ask for the specific record or field required to decide.
• Conflicting evidence: identify the disagreement and route it to an authorized reviewer.
• Unverified source or claim: withhold the unsupported conclusion until support can be checked.
• Action status unknown: report the unresolved status and check the destination before deciding whether to retry.
• Request outside policy: state the boundary and offer an allowed next step where available.

For example, a hypothetical expenses assistant should ask which policy version applies before asserting a reimbursement limit. If a payment response is missing, saying that payment succeeded would outrun the evidence. This is our proposed workflow behavior, not a tested product result.

Ground the first three outcomes in a citation and source check. Connect the last two to explicit action controls. A model’s statement of confidence is not the evidence record.

How should you score a test without hiding unanswered cases?

Define the eligible test set, authoritative reference and whole-answer acceptance rule first. Have reviewers label whether each case is answerable from that reference before evaluating the system. If they cannot establish answerability, keep the case and its label unresolved; do not quietly exclude it.

For this simple convention, an answer passes only when every material claim meets the declared reference. A substantive partial answer counts as an answer and is assessed under that rule. A clarification with no substantive answer counts as an abstention. For a more detailed claim-level analysis, use the measurement guide.

Keep these outcomes separate:

1. Delivered answer, reference check passed.
2. Delivered answer, reference check failed.
3. Delivered answer, reference check unresolved.
4. Abstention on an answerable case.
5. Abstention on an unanswerable case.
6. Abstention where answerability is unresolved.

Also record any tool or evaluation failure. Do not relabel a crash as a deliberate abstention; include it in the eligible-case denominator and report it separately.

What do coverage and answer error rate show?

Under our convention, answer coverage = delivered answers ÷ all eligible cases. When every delivered answer has a resolved reference judgment, answer error rate = failed answers ÷ delivered answers. Specify the acceptance rule: this is an error rate under that rule, not a universal definition of hallucination.

If some answers remain unjudged, report the pass, fail and unresolved counts. A failure rate among judged answers must use that explicit label and denominator; it must not hide the unjudged share. If no answer was delivered, answer error rate is undefined, not zero.

Hypothetical example, not a benchmark: 100 eligible cases contain 80 answerable and 20 unanswerable questions. A system delivers 70 answers: 65 pass the reference check and five fail it. It withholds answers on 15 answerable cases and 15 unanswerable cases. All judgments are resolved and there are no execution failures.

Its answer coverage is 70/100 = 70%. Its answer error rate is 5/70 = 7.1%. Its unnecessary-abstention rate is 15/80 = 18.75% of answerable cases; its appropriate-abstention rate is 15/20 = 75% of unanswerable cases. These figures describe different outcomes and should remain separate.

For the example, all 65 passing answers are on answerable cases and all five failing answers are on unanswerable cases. In a real test, an answerable question can also receive a wrong answer. Preserve the case-level labels so that difference remains visible.

What should happen after an abstention?

Give the unresolved case an owner, a reason and a permitted next step. Our suggested record contains the case ID, evidence version, answerability label, delivered response, answer/abstain/failure status, reference judgment, reason for withholding, reviewer and eventual resolution. Keep the automated outcome separate from the later human-assisted outcome.

When comparing confidence thresholds or handling rules, use the same declared test set and report all outcome counts. This guide recommends a reporting practice; it does not prescribe an optimal threshold or claim that an abstention feature guarantees accuracy. The survey’s accessible opening describes threshold methods but establishes no universal setting. Survey.

A hallucination-free claim needs this accounting. Ask both what unsupported output was prevented and how much useful work the system left unresolved.

What source material was read?

On October 13, 2026, direct opens returned no readable body. Exact indexed source text supplied the survey’s v3 header, abstract and initial introduction, plus the Abstain-QA abstract and metadata. Full papers and detailed metric sections were not available. The formulas, workflow rubric and arithmetic example above are explicitly our own reporting convention; no numerical study result is claimed.