AI confidence threshold: when to hold an LLM's answer

An AI confidence threshold is the rule that decides when a model's output is applied automatically and when a person has to look first. Most LLM features start without one, because in "suggest" mode a person reads everything anyway. The moment the feature starts acting on its own, you need the rule. On a compliance platform I work on, five stages of the lifecycle are AI-driven. Before this work, one of them had a lone hardcoded 60 that lit up a badge and stopped nothing, and the other four had no notion of confidence at all. This post covers the gate I built: the rule, where it sits, how thresholds are set, what "confidence" should actually mean, and how each decision is audited.

The rule fits in four lines

The gate itself is small. At or above the threshold, the stage proceeds. Below it, the result is held for a person. And a missing score is held too, because an AI that returned no confidence has not said it is confident.

def gate(confidence: float | None, threshold: float) -> str:
    if confidence is None:
        return "held"
    return "proceed" if confidence >= threshold else "held"

That third rule is the one people skip. It is tempting to treat a missing value as "no opinion" and let the result through. A missing score is more likely a symptom of something going wrong upstream than a neutral opinion: a parse failure, a prompt change, a field the model stopped returning. Those are exactly the cases you want a person to see.

Block, don't badge

A confidence badge in the UI is an annotation. It tells a reader the model was unsure, after the system has already acted on the answer. The old hardcoded 60 was this kind: visible and powerless.

The new gate blocks. Several consumers listen to the same event, and they run in registration order, so the gate is registered ahead of every consumer that acts on the result. A test fails if anyone flips that order. Without that test, a harmless-looking refactor of the registration list could quietly turn the gate back into a badge.

The same principle covers what the AI is allowed to do at all. The AI engine holds no permission and is refused at the approval gate. It writes analysis and proposals, and each one stays a proposal until a person accepts it. I wrote more about that boundary in AI agent guardrails in production.

How do you set an AI confidence threshold?

Set it per stage and per customer, because the cost of a wrong answer differs between steps and different organizations accept different risk. Start high, hold more than you need to, and lower the threshold for a stage only after reviewing what was held. Store the threshold used with every decision, so changing it later never rewrites history.

Per stage and per tenant matters in practice. A client who trusts the validator on document text may still want every treatment plan read by a person. One global number cannot express that.

There was also a second copy of the threshold inside the evidence validation code. With a tenant set to 90, one screen called a 65 fine while the gate held it. Removing the duplicate meant there is one number, owned by the gate, and every screen reads it.

Confidence should not be a model's opinion of itself

This is the part I would push hardest on. Asking an LLM how confident it is gives you a number, and that number is easy to log and hard to trust. In three stages of the gate, confidence is something measurable instead.

• Take the lowest, not the average. When validation checks several controls in one pass, the confidence reported is the lowest of them. The gate exists to catch the result nobody should act on unattended, and averaging hides it behind the ones that were sure.
• Count what was known. For risk scoring, confidence is the share of the six input factors that were actually known. A residual worked out from control counts alone scores low and reaches a person.
• Score agreement between attributes. For duplicate matching, confidence is built from which attributes agree. Wording alone lands at 35 and is held. Wording and subject together reach 70 and proceed. Two candidates tied at the top score zero, because a tie means the system cannot say which record it matched.

Each of these is a rule a reviewer can check by hand, and that makes the threshold meaningful. If you are still building the evaluation side, how to evaluate an LLM feature before you ship it covers how I test outputs before any gate exists.

Keep "inconclusive" apart from "failed"

Some results are neither a pass nor a fail. In the re-test stage, which was the fifth and last stage wired to the gate, three refusals are kept apart from a failed verdict: nothing was submitted as proof, no steps were imported, or the assessor was unreachable. These land as inconclusive.

Reporting them as failures would send work back to a person when nothing about their fix is known. An inconclusive result asks a different question ("can you give me something to check?") and should go to a different place.

Record every decision with the threshold applied

Each gate decision stores threshold_applied, the threshold in force at that moment. Moving a threshold later does not change what a past hold was measured against. Without it, an auditor looking at a hold from last month sees today's threshold and draws the wrong conclusion.

A partial unique index stops one subject waiting at the same gate twice. The shape, simplified:

CREATE UNIQUE INDEX one_open_hold_per_gate
  ON gate_decisions (stage, subject_id)
  WHERE outcome = 'held';

Postgres covers the mechanics in its docs on partial indexes.

The audit panel then shows the confidence beside each AI change and attributes it to the engine's service account, never to a person. A decision taken at 51 and one at 99 are different decisions, and the trail should say which one it was.

Where I'd start

1. List every place your LLM output is acted on without a person, and give each its own stage name.
2. Write the four-line rule, including "missing means held", and put it in front of every consumer that acts.
3. Store thresholds per stage and per customer, and keep exactly one copy.
4. Replace self-reported confidence with something countable where you can: worst case, known inputs, attribute agreement.
5. Log the threshold applied and the confidence with every decision, under the engine's own identity.

For the design question of where people belong in the loop at all, see human-in-the-loop AI design, and for schema checks on the output itself, validating LLM output before it reaches users.