What does calibrated confidence mean for an LLM classifier?

A classifier is calibrated when its stated probabilities match how often it is right: of all answers given with 0.8 confidence, about 80% are correct. The property holds on average across many answers, so any single answer can still be wrong. That is why a router uses confidence to decide which items a human should check.

How do I pick a confidence threshold for routing to a human?

Label a few dozen real items by hand, run them through the classifier, and look at where the correct and incorrect answers fall. Choose a threshold with margin on both sides, because individual probabilities can move between runs, and re-check it whenever the questions or the input format change.

Is the OpenRouter decisions API production-ready?

As of September 2026 the endpoint lives under /api/alpha/decisions and OpenRouter describes it as alpha, so the request or response schema may change. Wrap it behind one function in your code, log every call, and keep a fallback path such as a human review queue for when it fails or changes.

LLM Routing with Typed Decisions

Q: Can I use a normal chat model with JSON output as a classifier instead?

Yes. Structured output from a general model returns a well-formed typed answer and can include a reason, which helps with audits. What it doesn't give you by default is a calibrated probability; a confidence number the model writes into its JSON is generated text and should be validated against labelled data before you gate on it.

LLM routing is the point in a system where software decides which path an item takes: which queue a ticket lands in, which model handles a request, which review finding a human has to read first. Most teams implement it by asking a chat model a question and parsing the prose it writes back. That works until you run it a thousand times and notice that the same kind of item gets a different answer on Tuesday than it did on Monday.

This post describes a different shape for the same job: a typed decision, where the model returns one value from a set you defined in advance plus a confidence number, and never a sentence. I use TypeSafe's Jev model for this through OpenRouter's decisions API, so the examples are concrete, but the pattern (typed answers, explicit thresholds, low confidence routed to a human) applies to any classifier you put in front of a workflow.

What is a typed decision in LLM routing?

A typed decision is an answer whose type you fix before the model sees the input: one option from a closed set, a position on an ordered scale, or the probability that a statement is true. The caller never has to parse free text, and every answer can be compared with every other answer to the same question.

Jev, a model from TypeSafe served on OpenRouter, is built entirely around this idea. Its documentation describes it as a "structured decision model" that makes "fast, structured decisions for software" and states plainly that it "does not produce reasoning traces, explanations, or free-form text" and "is not a drop-in replacement for a chat model" (OpenRouter Jev docs). OpenRouter's explainer calls it a "System One" model, after Daniel Kahneman's fast, pattern-matching mode of thinking, and dates its early-access release to 15 September 2026 (What is Jev?).

Jev answers three kinds of question, which my skill file calls the three primitives:

Choice: one option from a fixed set. You map each option to a description of what it means, and the answer carries a probability for every option.
Score: a rating on ordered levels, written worst to best. OpenRouter's explainer caps a scale at ten levels. The answer is a probability-weighted position across the level indices.
Noul: the probability that a statement is true. There is no separate confidence field, because the probability is the answer.

The discipline this forces is the useful part. You can't ask "what do you think of this change?" You have to decide what the possible verdicts are, write down what each one means, and accept that the model will pick among them and tell you how sure it is.

How does the Jev decisions API work?

The caller sends one JSON request with a state (the thing being judged) and a map of named questions; the API returns a map of typed answers plus token usage and cost. The endpoint is POST https://openrouter.ai/api/alpha/decisions and, as the path says, it is an alpha API, so treat a schema change as possible.

As of September 2026 the model id is typesafe/jev-1.13, with a 32,000-token context and pricing of $0.042 per million input tokens and $0 for output (OpenRouter Jev docs). OpenRouter's explainer works through a three-question call of 447 input tokens that cost $0.000019, about two-thousandths of a cent. Only an OpenRouter key is needed, no separate TypeSafe account. There are no sampling parameters to tune.

This is a request with all three primitives, taken from my own skill file:

{
  "state": "<the thing being judged: a diff hunk, a ticket, a plan>",
  "questions": {
    "layer": {
      "type": "choice",
      "instructions": "Which layer does this defect belong to?",
      "criteria": {
        "domain": "Business rules and entities",
        "application": "Use-case orchestration",
        "infrastructure": "Persistence, HTTP, framework wiring"
      }
    },
    "risk": {
      "type": "score",
      "instructions": "How risky is this change to ship?",
      "criteria": [
        "Cosmetic, no behaviour change",
        "Behavioural but well covered by tests",
        "Behavioural with thin coverage",
        "Touches money, auth, or data integrity"
      ]
    },
    "crosses": {
      "type": "noul",
      "instructions": "This change introduces a cross-module dependency.",
      "criteria": {
        "true": "Reaches into another module's internals",
        "false": "Stays inside its module or goes through a facade"
      }
    }
  }
}

And the shape of what comes back:

{
  "answers": {
    "layer":   { "type": "choice", "choice": "application", "confidence": 0.75,
                 "probabilities": { "domain": 0.11, "application": 0.84, "infrastructure": 0.05 } },
    "risk":    { "type": "score", "score": 1.99, "confidence": 0.99,
                 "probabilities": { "0": 0, "1": 0.01, "2": 0.99 },
                 "legend": { "0": "...", "1": "...", "2": "..." } },
    "crosses": { "type": "noul", "noul": 0.96 }
  },
  "usage": { "input_tokens": 476, "output_tokens": 70, "cost": 0.00002 }
}

Two details matter when you read it. First, state can be a string, an object or an array, and passing structured context as an object works better than flattening it into a sentence. Second, a score is interpolated across level indices: 1.99 on a four-level scale sits just under level 2 ("behavioural with thin coverage"). Read it against the legend, never as a percentage.

Fan-out: one state goes into a single decisions call, which returns a Choice (category), a Score (severity) and a Noul (does a rule hold). A router then sorts the item using the answers and their confidence.

Fan out per item, not per question

Questions in one call don't interfere with each other, and you pay for the input once. So the rule in my skill is one fan-out call per item: pass the item as state and ask every question you might need about it in the same request. That costs the same as asking one question, cuts latency, and keeps the answers mutually consistent because they were all computed against the same input. Speculative extra questions are cheap enough to include just in case.

Why is confidence a second axis?

The answer tells you what the model picked; the confidence tells you whether to act on it without a person looking. A confident wrong answer and an unconfident right one look identical if you only read the answer field, so a router that drops the confidence throws away the most useful part of the response.

"Calibrated" has a precise meaning here. OpenRouter's explainer says that when Jev reports 0.8, it is right about that kind of answer about 80% of the time, but "only when you average across many answers. Any single answer can still be wrong." That is the standard definition of calibration used in machine learning, where modern neural networks are known to be over-confident unless calibrated explicitly (Guo et al., 2017). It means confidence is a property of a population of decisions, which is exactly how a router uses it.

The working rules in my skill file are short:

Act on high confidence, route low confidence. A floor such as confidence < 0.6 is a decision rule you can write down, review and change. Items below it go to a human queue, not to the bin.
A mid-range Noul means the question is bad. A probability between 0.3 and 0.7 usually says the statement was ambiguous. Sharpen the instructions or add true/false criteria rather than believing the number.
A probability is not permission. The model sorts and flags. Decisions a human owns stay with the human.

Routing bands: for Choice and Score answers, confidence below a floor (0.6 in this example) sends the item to a human and confidence above it lets the system act and log. For a Noul, 0.3 to 0.7 signals an ambiguous question to rewrite.

How to choose the threshold

Don't guess the floor; measure it. OpenRouter's own Jev tutorial labels a small sample of marketplace listings, looks at where the correct and incorrect answers fall, and picks a threshold with margin on both sides, "because individual probabilities move between runs." On its sample of 24 listings, 0.8 was the lowest threshold with zero wrong rejects (How to use Jev). The same method works for your data: a few dozen hand-labelled items, a threshold per question, and a re-check when the questions change. That labelled set is a small eval, and it belongs in the same place as your other evals for LLM features.

Where typed decisions fit: triage, risk scoring and routing

Typed decisions earn their place when the same judgement is applied to many items and drift between items would be a defect. If you'd write the same instructions twice, it's a candidate.

These are the uses in my own agent skills, which the skills workflow post describes in more detail:

Triaging review findings. Pass the diff hunk as state and ask, in one call, a Choice over the finding taxonomy, a Score for severity and Nouls for the repository's own rules (module boundaries, response envelope shape, raw query usage). Sort by severity, list the low-confidence findings separately as "needs a human look", and never let the result decide what ships. With agents producing more pull requests than people can read, this kind of sorting is one answer to the review bottleneck.
Risk scoring across a diff or a backlog. The same ordered scale applied to every file or every ticket. Hand-judging drifts most on exactly this kind of long, repetitive list.
Routing work. A Choice over the paths a task can take, with the confidence deciding whether the agent proceeds or asks.
Pressure-testing a plan. Pass the plan as state and ask whether it needs a new module, whether it crosses a module boundary and how large it is. The answers decide what to ask the human about, not what to build.

The routing step itself is a few lines. This is pseudo-code, not the helper I use:

# pseudo-code: sort review findings with one fan-out call each
for finding in findings:
    a = decide(state=finding.hunk, questions=REVIEW_QUESTIONS)
    ambiguous = 0.3 <= a["crosses"]["noul"] <= 0.7
    if a["category"]["confidence"] < FLOOR or ambiguous:
        needs_human.append(finding)
    else:
        sorted_findings.append((a["severity"]["score"], finding))
sorted_findings.sort(reverse=True)

The same shape works as a model router, sending easy requests to a small model and hard ones to a large model or a person. That use sits next to caching and batching in cutting LLM cost and latency.

Decision model or general LLM with structured output?

A general LLM with a strict output schema can also return a typed answer, and it can explain itself. A dedicated decision model returns a typed answer with probabilities, costs far less per call, and cannot explain anything. Which is better depends on whether you need the explanation or the number.

Structured output from a general model is mature. Claude's API enforces a JSON schema through output_config.format (Claude structured outputs), and the AI SDK has an Output.choice helper for enum selection (AI SDK structured data). What those give you is a well-formed answer. What they don't give you by default is a calibrated probability: a confidence number the model writes into its JSON is generated text, and I'd treat it as uncalibrated until I had measured it against labels.

Criterion	Decision model (Jev)	General LLM + structured output
Output	Choice, Score or Noul only	Any JSON schema, plus prose if wanted
Explanation	None, by design	Available, useful for audit trails
Confidence	Probabilities per option and a confidence field	Not native; self-reported numbers need validating
Price (Sept 2026)	$0.042/M input, $0 output	Input and output both billed
Context	32,000 tokens	Up to about 1M tokens on current large models
Tuning knobs	No sampling parameters	Temperature, prompts, examples
API maturity	Alpha endpoint	Generally available
Best for	High-volume, repeated judgements with a threshold	Low-volume calls that need reasons or free-form fields

The two combine well. Let the decision model sort a thousand items, then ask a general model to explain the twenty it was unsure about, or hand those twenty to a person.

When not to use a decision model

Don't use a typed decision model for anything that needs text, for one-off calls the context already answers, or for decisions a human is accountable for. Its honest limits are part of its design.

No prose, no code, no reasons. It physically can't produce them. If the output has to be read by someone who will ask "why?", you need another component.
One-off judgements. A network round-trip is not an improvement on reading the file in front of you. The value appears with volume and repetition.
It sees only the state you pass. No repository access, no memory between calls, no way to look anything up. A vague state yields a confident answer to the wrong question, and nothing in the response will tell you.
Numbers look authoritative either way. Garbage in is specifically dangerous with numeric output, because a 0.93 reads as trustworthy whether the input made sense or not.
Alpha API. The endpoint path says alpha. Wrap it behind your own interface so a schema change is a one-file fix, and keep a fallback path (a human queue is fine) for when it fails.
Architecture and design calls. Use the model to decide what to ask the owner about, then ask them.

Practical checklist for typed LLM routing

Write the closed set first. Options with a one-line meaning each, or ordered levels worst to best.
Pass structured state. An object with named fields, not a paragraph that summarizes them.
One fan-out call per item. Ask every question you might need in the same request.
Label a few dozen items and pick each threshold with margin on both sides.
Route, don't drop. Low confidence and mid-range Nouls go to a human queue with the item attached.
Rewrite ambiguous questions instead of trusting a 0.5.
Log every decision with its confidence and spot-check the auto-applied ones.
Wrap the alpha API behind one function with a fallback path.
Keep ownership with people. A probability is not permission.

If you're building triage or routing into an agent workflow and want a second pair of eyes on it, see AI engineering & MCP servers.

Typed decisions for LLM routing and triage: calibrated confidence with Jev

Key takeaways