[{"data":1,"prerenderedAt":774},["ShallowReactive",2],{"blog-reasoning-models-effort-thinking-budget-en":3},{"slug":4,"published":5,"minutes":6,"category":7,"tags":8,"keywords":14,"about":26,"sources":36,"cover":61,"og":62,"expertise":63,"locales":64,"lang":65,"title":68,"description":69,"coverAlt":70,"metaTitle":71,"takeaways":72,"faq":78,"toc":94,"blocks":125,"others":540},"reasoning-models-effort-thinking-budget","2026-10-17",10,"llmops",[9,10,11,12,13],"Reasoning models","Effort","LLMOps","Cost","Evals",[15,16,17,18,19,20,21,22,23,24,25],"reasoning effort parameter","claude effort parameter","claude adaptive thinking","budget_tokens deprecated","openai reasoning effort","gemini thinking_level","thinking tokens cost","reasoning tokens billing","task budget claude","overthinking reasoning models","max_tokens thinking",[27,30,33],{"name":28,"url":29},"Reasoning language model","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FReasoning_language_model",{"name":31,"url":32},"Large language model","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FLarge_language_model",{"name":34,"url":35},"Prompt engineering","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FPrompt_engineering",[37,40,43,46,49,52,55,58],{"title":38,"url":39},"Claude Platform Docs: Effort","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Feffort",{"title":41,"url":42},"Claude Platform Docs: Extended thinking (manual budget_tokens mode)","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fextended-thinking",{"title":44,"url":45},"Claude Platform Docs: Steering thinking (cost, caching, pricing)","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fthinking-steering-and-cost",{"title":47,"url":48},"Claude Platform Docs: Task budgets","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Ftask-budgets",{"title":50,"url":51},"OpenAI API Docs: Reasoning models","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Freasoning",{"title":53,"url":54},"Google AI for Developers: Gemini thinking","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fthinking",{"title":56,"url":57},"Chen et al.: Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs (arXiv 2412.21187)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2412.21187",{"title":59,"url":60},"Gema et al.: Inverse Scaling in Test-Time Compute (arXiv 2507.14417)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2507.14417","\u002Fimages\u002Fblog\u002Freasoning-models-effort-thinking-budget\u002Fcover.webp","\u002Fimages\u002Fblog\u002Freasoning-models-effort-thinking-budget\u002Fog.jpg","ai-engineer",[65,66,67],"en","de","hu","Reasoning effort and thinking budgets: how much should a model think?","How effort, adaptive thinking and thinking levels work on Claude, OpenAI and Gemini, what thinking tokens cost, and how to pick a level with an effort sweep.","Four upright bars rising from short to tall on one baseline, with a small marker above the tallest.","Reasoning effort and thinking budgets · Balázs Csorba",[73,74,75,76,77],"Claude, OpenAI and Gemini all moved from token budgets to named levels: effort on Claude and OpenAI, thinking_level on Gemini.","Thinking tokens are billed as output tokens even when you only see a summary or nothing at all.","max_tokens is the only hard cap on a request; effort and task budgets are guidance the model can exceed or ignore.","More thinking is not always better: research shows overthinking on easy problems and accuracy dropping on some tasks as reasoning grows.","Pick the level with an effort sweep on your own evals, then keep it stable per conversation so you do not break the prompt cache.",[79,82,85,88,91],{"q":80,"a":81},"What is the difference between effort and a thinking budget?","A thinking budget, such as budget_tokens on older Claude models, sets a token target for reasoning before the answer. Effort is a behavioural level that shapes all output tokens, including text and tool calls, and lets the model decide whether to think at all. On Claude 4.7 and later models budget_tokens is rejected with a 400 error and effort is the control.",{"q":83,"a":84},"Are thinking tokens billed if I cannot see them?","Yes. Anthropic bills the full thinking tokens as output tokens whether display is summarized or omitted. OpenAI bills hidden reasoning tokens as output tokens, and Gemini prices a thinking response as the sum of output and thinking tokens.",{"q":86,"a":87},"Which effort level should I start with?","Start with the model default and run a sweep on your own eval set. Defaults differ: Claude Opus 5.5 and Haiku 5.5 default to medium, most other Claude models to high, and gpt-5.5 to medium. Step down where quality holds and step up only where your evals show a gain.",{"q":89,"a":90},"Why did my response come back empty or truncated?","The model hit max_tokens or max_output_tokens while still thinking. Claude returns stop_reason max_tokens, OpenAI returns status incomplete with reason max_output_tokens, and Gemini returns incomplete output. Raise the cap or lower the effort level.",{"q":92,"a":93},"Does changing effort break prompt caching?","On Claude, yes for top-level changes: the effort value is rendered into the prompt, so a new value invalidates cache breakpoints. Several current Claude models support a per-message effort change in beta that keeps the cached prefix intact.",[95,98,101,104,107,110,113,116,119,122],{"id":96,"title":97},"what-you-pay-for","What thinking tokens are and what you pay for",{"id":99,"title":100},"three-apis","The controls on Claude, OpenAI and Gemini",{"id":102,"title":103},"from-budget-tokens","From budget_tokens to effort",{"id":105,"title":106},"max-tokens-hard-cap","max_tokens is the only hard cap",{"id":108,"title":109},"more-is-not-better","More thinking is not always better",{"id":111,"title":112},"effort-sweep","Picking a level: the effort sweep",{"id":114,"title":115},"per-request-routing","Routing effort per request without breaking the cache",{"id":117,"title":118},"agent-loops","Agent loops: interleaved thinking and task budgets",{"id":120,"title":121},"measure-it","Measure thinking like any other cost",{"id":123,"title":124},"sources","Sources",[126,139,146,149,152,167,168,171,262,286,307,308,318,320,327,328,346,354,356,357,360,374,375,378,387,389,409,410,424,439,442,443,456,479,480,503,510,511],{"type":127,"content":128},"paragraph",[129,130,134,135,138],"A year ago, controlling how much a model reasoned meant picking a number: a thinking budget of 4,000 or 16,000 tokens. In 2026 the three big APIs have converged on something different. Claude and OpenAI expose an ",{"tag":131,"children":132},"code",[133],"effort"," level, Gemini a ",{"tag":131,"children":136},[137],"thinking_level",", and the model decides per request whether to think at all. That is easier to configure and harder to reason about, because the knob no longer maps to a token count.",{"type":127,"content":140},[141,142,145],"This post covers what each API actually offers, what thinking costs on the invoice, why ",{"tag":131,"children":143},[144],"max_tokens"," is the only hard limit you have, why more thinking can make answers worse, and how I pick a level for a feature: an effort sweep on a real eval set, then a fixed setting per conversation so the prompt cache survives.",{"type":147,"level":148,"id":96,"text":97},"heading",2,{"type":127,"content":150},[151],"Reasoning models generate tokens before, and on current models also between, their visible answer and tool calls. On every major API those tokens are billed as output, the expensive side of the price sheet. OpenAI says plainly that reasoning tokens are not visible through the API but occupy context and are billed as output tokens. Gemini prices a thinking response as the sum of output and thinking tokens. Anthropic bills the full internal thinking even when the response only carries a summary.",{"type":127,"content":153},[154,155,158,159,162,163,166],"On Claude, the ",{"tag":131,"children":156},[157],"display"," setting changes only what you see. With ",{"tag":131,"children":160},[161],"summarized"," you get summary text, with ",{"tag":131,"children":164},[165],"omitted"," the thinking field is empty, and in both cases the billed output tokens are the full thinking the model generated. Anthropic warns that the billed count does not match the visible count. There is a second, quieter cost: Claude Opus 4.5 and models numbered 4.6 and higher keep thinking blocks from earlier turns in context and bill them as input on every following request.",{"type":147,"level":148,"id":99,"text":100},{"type":127,"content":169},[170],"The three APIs look similar from a distance and differ in the details that break code: where the parameter lives, which values a model accepts and what the default is. The defaults matter most, because an omitted parameter is a decision too.",{"type":172,"head":173,"rows":182},"table",[174,176,178,180],[175],"API",[177],"Parameter",[179],"Values",[181],"Default",[183,214,243],[184,186,190,207],[185],"Claude Messages API",[187],{"tag":131,"children":188},[189],"output_config.effort",[191,194,195,194,198,194,201,194,204],{"tag":131,"children":192},[193],"low",", ",{"tag":131,"children":196},[197],"medium",{"tag":131,"children":199},[200],"high",{"tag":131,"children":202},[203],"xhigh",{"tag":131,"children":205},[206],"max",[208,210,211,213],{"tag":131,"children":209},[197]," on Opus 5.5 and Haiku 5.5, ",{"tag":131,"children":212},[200]," on most others",[215,217,221,238],[216],"OpenAI Responses API",[218],{"tag":131,"children":219},[220],"reasoning.effort",[222,194,225,194,228,194,230,194,232,194,234,194,236],{"tag":131,"children":223},[224],"none",{"tag":131,"children":226},[227],"minimal",{"tag":131,"children":229},[193],{"tag":131,"children":231},[197],{"tag":131,"children":233},[200],{"tag":131,"children":235},[203],{"tag":131,"children":237},[206],[239,240,242],"Model-dependent, ",{"tag":131,"children":241},[197]," on gpt-5.5",[244,246,249,258],[245],"Gemini API",[247],{"tag":131,"children":248},[137],[250,194,252,194,254,194,256],{"tag":131,"children":251},[227],{"tag":131,"children":253},[193],{"tag":131,"children":255},[197],{"tag":131,"children":257},[200],[239,259,261],{"tag":131,"children":260},[197]," on gemini-3.8-flash",{"type":127,"content":263},[264,265,267,268,270,271,273,274,276,277,279,280,282,283,285],"Not every model accepts every value. On Claude, not every model that supports ",{"tag":131,"children":266},[206]," supports ",{"tag":131,"children":269},[203],", and the effort docs carry per-model recommendations that override the general table: Opus 4.7 and 4.8 recommend starting at ",{"tag":131,"children":272},[203]," for coding and agents, while Sonnet 5.5 suggests ",{"tag":131,"children":275},[197]," for well-specified agentic coding. On OpenAI, GPT-6 Astra rejects ",{"tag":131,"children":278},[224]," with HTTP 400. On Gemini, gemini-3-pro-preview only knows ",{"tag":131,"children":281},[193]," and ",{"tag":131,"children":284},[200],".",{"type":287,"variant":288,"title":289,"body":290},"callout","warn","An omitted effort is not a stable setting",[291],[292,293,295,296,298,299,301,302,285],"Claude Opus 5.5 defaults to ",{"tag":131,"children":294},[197],", while Claude Opus 5 defaulted to ",{"tag":131,"children":297},[200],". Anthropic notes that a request which omits ",{"tag":131,"children":300},[133]," therefore runs one level lower than it did on Opus 5. If you swap model ids without setting effort explicitly, you change two variables at once. Pin the level in code, and treat a model upgrade as a migration with its own ",{"tag":303,"to":304,"children":305},"link","\u002Fblog\u002Fllm-model-migration-regression-testing",[306],"regression run",{"type":147,"level":148,"id":102,"text":103},{"type":127,"content":309},[310,311,314,315,317],"The old Claude control, ",{"tag":131,"children":312},[313],"thinking: {type: \"enabled\", budget_tokens: N}",", is deprecated on the 4.6 models and rejected with a 400 error on Claude 4.7 and later. It still matters if you run Claude 4.5 or earlier, where it is the only thinking mode: the budget must be at least 1,024 tokens and below ",{"tag":131,"children":316},[144],", and Anthropic recommends batch processing for budgets above 32k to avoid timeouts. The migration is small in code and large in behaviour.",{"type":131,"code":319},"\u002F\u002F before: fixed budget, Claude thinks on every request\n{ \"model\": \"claude-sonnet-4-6\", \"max_tokens\": 16000,\n  \"thinking\": { \"type\": \"enabled\", \"budget_tokens\": 10000 } }\n\n\u002F\u002F after: adaptive thinking, depth controlled by effort\n{ \"model\": \"claude-sonnet-4-6\", \"max_tokens\": 16000,\n  \"thinking\": { \"type\": \"adaptive\" },\n  \"output_config\": { \"effort\": \"high\" } }",{"type":127,"content":321},[322,323,326],"With a fixed budget, Claude thinks on every request. With adaptive thinking it decides per request, and at lower effort it may skip thinking entirely on easy inputs, so do not write code that assumes every assistant turn starts with a thinking block. Effort also reaches beyond thinking: it shapes text and tool calls, and lower levels make fewer and terser tool calls. On Opus 5.5, adaptive thinking cannot be switched off at all; ",{"tag":131,"children":324},[325],"thinking: {type: \"disabled\"}"," returns a 400.",{"type":147,"level":148,"id":105,"text":106},{"type":127,"content":329},[330,331,333,334,337,338,341,342,345],"Effort is guidance, not a budget. Anthropic calls it a behavioural signal, and the hard ceiling is ",{"tag":131,"children":332},[144],", which covers thinking and answer together. If a hard request thinks long, it can run into that ceiling before writing the answer. Claude then stops with ",{"tag":131,"children":335},[336],"stop_reason: \"max_tokens\"",". OpenAI returns ",{"tag":131,"children":339},[340],"status: \"incomplete\""," with reason ",{"tag":131,"children":343},[344],"max_output_tokens",", possibly before any visible output, and you still pay for the reasoning. Gemini behaves the same way and bills the thinking tokens it generated.",{"type":127,"content":347},[348,349,282,351,353],"OpenAI recommends reserving at least 25,000 tokens for reasoning and output when you start, and Anthropic suggests starting around 64k for ",{"tag":131,"children":350},[203],{"tag":131,"children":352},[206]," on its larger models. When you see truncation, decide which fix applies: if the truncated requests needed the reasoning, raise the cap; if they were overthought, lower the effort. Either way, handle the truncated case in code instead of shipping a half answer.",{"type":131,"code":355},"\u002F\u002F Claude: treat truncation as a failure, not as an answer\nconst res = await client.messages.create({\n  model: \"claude-opus-5-5\",\n  max_tokens: 32000,\n  output_config: { effort: \"high\" },\n  messages,\n});\nif (res.stop_reason === \"max_tokens\") {\n  metrics.increment(\"llm.truncated\", { effort: \"high\" });\n  \u002F\u002F retry once with a larger cap, or fall back to a lower effort\n}\nconst thinking = res.usage.output_tokens_details?.thinking_tokens ?? 0;",{"type":147,"level":148,"id":108,"text":109},{"type":127,"content":358},[359],"The intuitive model, more reasoning means better answers, holds on hard problems and fails at the edges. Chen and colleagues documented overthinking in o1-style models in late 2024: excessive compute spent on simple problems with minimal benefit, such as long chains for 2+3. In July 2025, Anthropic-affiliated researchers published tasks where longer reasoning lowers accuracy, an inverse scaling between test-time compute and quality, including models distracted by irrelevant information and drifting toward spurious correlations.",{"type":127,"content":361},[362,363,365,366,368,369,373],"The vendor docs say the same in operational terms. Anthropic's guidance for Opus 4.7 notes that ",{"tag":131,"children":364},[206]," adds significant cost for small gains on most workloads and can lead to overthinking on structured-output or less intelligence-sensitive tasks. OpenAI recommends ",{"tag":131,"children":367},[203]," only when evals show a clear benefit. For an extraction or classification feature, the right answer is often the lowest level that holds quality, combined with ",{"tag":303,"to":370,"children":371},"\u002Fblog\u002Fstructured-outputs-json-schema-llm",[372],"structured outputs"," so the format does not depend on how long the model thought.",{"type":147,"level":148,"id":111,"text":112},{"type":127,"content":376},[377],"Every vendor ends its guidance with the same instruction: test on your own workload. Anthropic explicitly says to run a fresh sweep instead of carrying settings over from an earlier model, because levels are recalibrated between generations. In practice that is a small script: the same eval set, the same model, every supported level, and four numbers per level.",{"type":379,"attrs":380,"inner":384,"caption":385},"diagram",{"viewBox":381,"role":382,"aria-labelledby":383},"0 0 720 170","img","rme1-t rme1-d","\u003Ctitle id=\"rme1-t\">Effort sweep workflow\u003C\u002Ftitle>\u003Cdesc id=\"rme1-d\">Run the same eval set at every effort level, record quality, cost, latency and truncation, choose the lowest level that meets the quality bar, pin it in code, and repeat on every model upgrade.\u003C\u002Fdesc>\u003Cdefs>\u003Cmarker id=\"rme1-ah\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\">\u003Cpath d=\"M0 0L10 5L0 10z\" class=\"d-head\" \u002F>\u003C\u002Fmarker>\u003C\u002Fdefs>\u003Crect x=\"20\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-box\" \u002F>\u003Ctext x=\"79\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">Eval set\u003C\u002Ftext>\u003Ctext x=\"79\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">real traffic\u003C\u002Ftext>\u003Cpath d=\"M138 85H158\" class=\"d-line\" marker-end=\"url(#rme1-ah)\" \u002F>\u003Crect x=\"160\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-sky\" \u002F>\u003Ctext x=\"220\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">Sweep\u003C\u002Ftext>\u003Ctext x=\"220\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">low to max\u003C\u002Ftext>\u003Cpath d=\"M279 85H299\" class=\"d-line\" marker-end=\"url(#rme1-ah)\" \u002F>\u003Crect x=\"301\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-gold\" \u002F>\u003Ctext x=\"360\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">Measure\u003C\u002Ftext>\u003Ctext x=\"360\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">score, cost, p95\u003C\u002Ftext>\u003Cpath d=\"M419 85H439\" class=\"d-line\" marker-end=\"url(#rme1-ah)\" \u002F>\u003Crect x=\"441\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-mint\" \u002F>\u003Ctext x=\"500\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">Choose\u003C\u002Ftext>\u003Ctext x=\"500\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">lowest that holds\u003C\u002Ftext>\u003Cpath d=\"M560 85H580\" class=\"d-line\" marker-end=\"url(#rme1-ah)\" \u002F>\u003Crect x=\"582\" y=\"47\" width=\"118\" height=\"76\" rx=\"10\" class=\"d-box\" \u002F>\u003Ctext x=\"641\" y=\"81\" text-anchor=\"middle\" class=\"d-text\">Pin\u003C\u002Ftext>\u003Ctext x=\"641\" y=\"103\" text-anchor=\"middle\" class=\"d-small\">per feature\u003C\u002Ftext>",[386],"The sweep is cheap compared with running the wrong level in production for a month. Rerun it whenever the model id changes.",{"type":131,"code":388},"for (const effort of [\"low\", \"medium\", \"high\", \"xhigh\"]) {\n  const runs = await Promise.all(evalSet.map(async (c) => {\n    const t0 = Date.now();\n    const r = await client.messages.create({\n      model, max_tokens: 64000, output_config: { effort },\n      messages: c.messages,\n    });\n    return { ms: Date.now() - t0, out: r.usage.output_tokens,\n             truncated: r.stop_reason === \"max_tokens\",\n             score: await grade(c, r) };\n  }));\n  report(effort, runs); \u002F\u002F pass rate, mean output tokens, p95 ms, truncation rate\n}",{"type":127,"content":390},[391,392,394,395,397,398,400,401,403,404,408],"Read the result as a curve, not a winner. If ",{"tag":131,"children":393},[197]," passes the same cases as ",{"tag":131,"children":396},[200]," at a fraction of the output tokens, ",{"tag":131,"children":399},[197]," wins. If ",{"tag":131,"children":402},[200]," fixes the cases that matter to the business, the extra tokens are cheap. The grading is the hard part, and it is the same work as any ",{"tag":303,"to":405,"children":406},"\u002Fblog\u002Fllm-evals-for-product-features",[407],"LLM eval for a product feature",": labelled cases from real traffic and a grader you trust.",{"type":147,"level":148,"id":114,"text":115},{"type":127,"content":411},[412,413,415,416,418,419,421,422,285],"One level per feature is the baseline. The next step is to vary it per request: ",{"tag":131,"children":414},[193]," for a routine confirmation, ",{"tag":131,"children":417},[200]," for the planning step. Anthropic lists dynamic effort as a best practice, with a catch. The resolved effort value is rendered into the prompt, so changing the top-level value between requests invalidates cache breakpoints. Anthropic's own example shows a full cache read on the second request and a fresh cache write on the third, after effort dropped from ",{"tag":131,"children":420},[197]," to ",{"tag":131,"children":423},[193],{"type":127,"content":425},[426,427,431,432,435,436,438],"Two ways around it. Across workloads, route at the feature boundary, where conversations do not share a cache anyway; the routing logic is the same as ",{"tag":303,"to":428,"children":429},"\u002Fblog\u002Fllm-cost-latency-prompt-caching-routing",[430],"model routing",". Inside a conversation, current Claude models such as Opus 5.5, Sonnet 5.5 and Haiku 5.5 support a per-message effort change in beta, behind the ",{"tag":131,"children":433},[434],"mid-conversation-output-config-2026-07-01"," header: an empty system message carrying ",{"tag":131,"children":437},[189]," changes the level from the next user turn and keeps the earlier prefix cached.",{"type":127,"content":440},[441],"Prompt-based steering is the third lever. Anthropic documents phrases such as \"Answer directly without deliberating.\" for routine turns, but warns that it is wording-sensitive and less reliable than effort, and on Haiku 5.5 the discouraging phrase did not reliably reduce thinking. I use it only after the level is right.",{"type":147,"level":148,"id":117,"text":118},{"type":127,"content":444},[445,446,448,449,451,452,455],"In an agent, one user message becomes many requests, and ",{"tag":131,"children":447},[144]," only caps each request, not the whole turn. Current models think between tool calls automatically. Anthropic notes that the first request after new user input typically carries most of the reasoning, and follow-ups that only process tool results can skip thinking even at ",{"tag":131,"children":450},[203],". OpenAI asks you to pass reasoning items back with function call outputs, through ",{"tag":131,"children":453},[454],"previous_response_id"," or the full output items, so the model can continue its reasoning without redoing it.",{"type":127,"content":457},[458,459,462,463,466,467,470,471,474,475,285],"For the whole loop, Claude has task budgets in beta (",{"tag":131,"children":460},[461],"task-budgets-2026-03-13","): ",{"tag":131,"children":464},[465],"output_config.task_budget"," with ",{"tag":131,"children":468},[469],"type: \"tokens\""," and a ",{"tag":131,"children":472},[473],"total"," of at least 20,000. The model sees a server-side countdown and paces itself to finish gracefully. It is advisory, not enforced, and a budget that is clearly too small can make Claude scope the task down or decline it. Anthropic suggests sizing it from the p99 of measured per-task spend. Effort tunes depth per step; the task budget tunes total breadth. The same agent-loop mechanics are covered in ",{"tag":303,"to":476,"children":477},"\u002Fblog\u002Fagent-loop-explained",[478],"the agent loop, explained",{"type":147,"level":148,"id":120,"text":121},{"type":127,"content":481},[482,483,486,487,490,491,494,495,498,499,502],"All three APIs report thinking separately, so there is no excuse for guessing. Claude returns ",{"tag":131,"children":484},[485],"usage.output_tokens_details.thinking_tokens",", which is always less than or equal to ",{"tag":131,"children":488},[489],"output_tokens"," and appears on the final ",{"tag":131,"children":492},[493],"message_delta"," event when streaming. OpenAI reports ",{"tag":131,"children":496},[497],"output_tokens_details.reasoning_tokens",". Gemini reports ",{"tag":131,"children":500},[501],"total_thought_tokens",". Put the field on every trace next to the effort level and model id.",{"type":127,"content":504},[505,506,285],"Three charts are enough: thinking tokens per request by feature, truncation rate by effort level, and quality score by effort level from your sampled evals. When thinking per request creeps up after a prompt change, you see it the same day instead of on the invoice. The tracing side is in ",{"tag":303,"to":507,"children":508},"\u002Fblog\u002Fagent-observability-opentelemetry",[509],"observability for LLM agents with OpenTelemetry",{"type":147,"level":148,"id":123,"text":124},{"type":512,"ordered":513,"items":514},"list",true,[515,519,522,525,528,531,534,537],[516],{"tag":517,"href":39,"children":518},"a",[38],[520],{"tag":517,"href":42,"children":521},[41],[523],{"tag":517,"href":45,"children":524},[44],[526],{"tag":517,"href":48,"children":527},[47],[529],{"tag":517,"href":51,"children":530},[50],[532],{"tag":517,"href":54,"children":533},[53],[535],{"tag":517,"href":57,"children":536},[56],[538],{"tag":517,"href":60,"children":539},[59],[541,597,664,721],{"slug":542,"published":5,"minutes":543,"category":7,"tags":544,"keywords":550,"about":564,"sources":572,"cover":591,"og":592,"expertise":63,"locales":593,"lang":65,"title":594,"description":595,"coverAlt":596},"deepseek-v4-1-flash-1017",15,[545,546,547,548,549],"DeepSeek","Open-weight models","KV cache","Mixture of experts","LLM cost",[551,552,553,554,555,556,557,558,559,560,561,562,563],"DeepSeek V4.1 Flash 1017","DeepSeek V4.1 Flash","Causal Encoder-Decoder","Compressed Sparse Attention 2","CSA2","FP4 KV cache","KV cache compression","SWA Bounded Replay","Engram conditional memory","DSpark speculative decoding","DeepSeek pricing","DeepSWE benchmark","open-weight coding model",[565,567,569],{"name":545,"url":566},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FDeepSeek",{"name":548,"url":568},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FMixture_of_experts",{"name":570,"url":571},"Transformer (deep learning architecture)","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FTransformer_(deep_learning_architecture)",[573,576,579,582,585,588],{"title":574,"url":575},"DeepSeek-AI: DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression (arXiv:2609.19969)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.19969",{"title":577,"url":578},"Hugging Face: deepseek-ai\u002FDeepSeek-V4.1-Flash-1017 model card","https:\u002F\u002Fhuggingface.co\u002Fdeepseek-ai\u002FDeepSeek-V4.1-Flash-1017",{"title":580,"url":581},"OpenRouter: DeepSeek V4.1 Flash 1017, pricing and providers","https:\u002F\u002Fopenrouter.ai\u002Fdeepseek\u002Fdeepseek-v4.1-flash-1017",{"title":583,"url":584},"The Decoder: Deepseek V4.1 Flash 1017 beats larger rivals in agentic tasks while once again massively cutting per-task costs","https:\u002F\u002Fthe-decoder.com\u002Fdeepseek-v4-1-flash-1017-beats-larger-rivals-in-agentic-tasks-while-once-again-massively-cutting-per-task-costs\u002F",{"title":586,"url":587},"AI Weekly: DeepSeek opens V4.1-Flash-1017 weights and $0.75 Search API","https:\u002F\u002Faiweekly.co\u002Falerts\u002Fdeepseek-opens-v41-flash-1017-weights-and-075-search-api",{"title":589,"url":590},"DeepSeek on X: DeepSeek-V4.1-Flash-1017 announcement","https:\u002F\u002Fx.com\u002Fdeepseek_ai\u002Fstatus\u002F2111375640717033777","\u002Fimages\u002Fblog\u002Fdeepseek-v4-1-flash-1017\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fdeepseek-v4-1-flash-1017\u002Fog.jpg",[65,66,67],"DeepSeek V4.1 Flash 1017: 890 bytes of KV cache per token, and what that buys an agent","A technical deep dive into DeepSeek V4.1 Flash 1017: the encoder-decoder split, CSA2 cache sharing, FP4 KV cache, the benchmarks and the real cost per agent task.","A tall stack of thin cards on the left, an arrow, and the same cards pressed into a stack a quarter of the height on the right.",{"slug":598,"published":5,"minutes":6,"category":7,"tags":599,"keywords":605,"about":617,"sources":627,"cover":658,"og":659,"expertise":63,"locales":660,"lang":65,"title":661,"description":662,"coverAlt":663},"llm-rate-limits-retries-provider-fallback",[600,601,602,603,604],"Rate limits","Retries","Fallback","Reliability","LLM gateway",[606,607,608,609,610,611,612,613,614,615,616],"llm rate limit handling","429 too many requests llm api","exponential backoff with jitter","anthropic 529 overloaded error","openai rate limit retry","llm provider fallback","litellm fallbacks","retry-after header","llm circuit breaker","multi provider llm failover","claude api rate limits",[618,621,624],{"name":619,"url":620},"Token bucket","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FToken_bucket",{"name":622,"url":623},"Exponential backoff","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FExponential_backoff",{"name":625,"url":626},"Circuit breaker design pattern","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FCircuit_breaker_design_pattern",[628,631,634,637,640,643,646,649,652,655],{"title":629,"url":630},"Claude Platform Docs: Rate limits","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fapi\u002Frate-limits",{"title":632,"url":633},"Claude Platform Docs: API errors","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fapi\u002Ferrors",{"title":635,"url":636},"OpenAI API Docs: Rate limits","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Frate-limits",{"title":638,"url":639},"AWS Builders Library: Timeouts, retries and backoff with jitter","https:\u002F\u002Fbuilder.aws.com\u002Fcontent\u002F3EumjoZascWd1oZiEgL8ORlv3qE\u002Ftimeouts-retries-and-backoff-with-jitter",{"title":641,"url":642},"MDN: Retry-After header","https:\u002F\u002Fdeveloper.mozilla.org\u002Fen-US\u002Fdocs\u002FWeb\u002FHTTP\u002FReference\u002FHeaders\u002FRetry-After",{"title":644,"url":645},"Martin Fowler: Circuit Breaker (2014)","https:\u002F\u002Fmartinfowler.com\u002Fbliki\u002FCircuitBreaker.html",{"title":647,"url":648},"LiteLLM Docs: Router, retries and cooldowns","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Frouting",{"title":650,"url":651},"LiteLLM Docs: Proxy fallbacks and reliability","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Fproxy\u002Freliability",{"title":653,"url":654},"OpenRouter Docs: Model fallbacks","https:\u002F\u002Fopenrouter.ai\u002Fdocs\u002Fguides\u002Frouting\u002Fmodel-fallbacks",{"title":656,"url":657},"AI SDK Docs: Settings (maxRetries, timeout)","https:\u002F\u002Fai-sdk.dev\u002Fdocs\u002Fai-sdk-core\u002Fsettings","\u002Fimages\u002Fblog\u002Fllm-rate-limits-retries-provider-fallback\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-rate-limits-retries-provider-fallback\u002Fog.jpg",[65,66,67],"LLM rate limits in production: retries, backoff and provider fallback","How LLM rate limits work, which errors to retry, backoff with jitter, retrying in one layer only, and fallback chains that do not break your output.","Small squares climbing three steps up to a wide rectangle, two more rectangles behind it in a row.",{"slug":665,"published":5,"minutes":666,"category":7,"tags":667,"keywords":671,"about":682,"sources":690,"cover":715,"og":716,"expertise":63,"locales":717,"lang":65,"title":718,"description":719,"coverAlt":720},"prompt-versioning-management",8,[668,669,11,13,670],"Prompt management","Versioning","Langfuse",[672,673,674,675,676,677,678,679,680,681],"prompt versioning","prompt management","prompt registry","version control for prompts","langfuse prompt management","mlflow prompt registry","prompts in git","openai prompt object deprecation","prompt rollback production","prompt ci evaluation",[683,684,687],{"name":34,"url":35},{"name":685,"url":686},"Version control","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FVersion_control",{"name":688,"url":689},"Continuous integration","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FContinuous_integration",[691,694,697,700,703,706,709,712],{"title":692,"url":693},"OpenAI API Docs: Migrate from prompt objects","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fprompting\u002Fmigrate-from-prompt-object",{"title":695,"url":696},"OpenAI API Docs: Prompt engineering (reusable prompts deprecation note)","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fprompt-engineering",{"title":698,"url":699},"Langfuse Docs: Prompt version control","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Fprompt-version-control",{"title":701,"url":702},"Langfuse Docs: Prompt caching in the SDKs","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Fcaching",{"title":704,"url":705},"Langfuse Docs: Link prompts to traces","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Flink-to-traces",{"title":707,"url":708},"MLflow Docs: Prompt Registry","https:\u002F\u002Fmlflow.org\u002Fdocs\u002Flatest\u002Fgenai\u002Fprompt-registry\u002F",{"title":710,"url":711},"Promptfoo Docs: GitHub Action","https:\u002F\u002Fwww.promptfoo.dev\u002Fdocs\u002Fintegrations\u002Fgithub-action\u002F",{"title":713,"url":714},"Claude Platform Docs: Prompt caching","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fprompt-caching","\u002Fimages\u002Fblog\u002Fprompt-versioning-management\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fprompt-versioning-management\u002Fog.jpg",[65,66,67],"Prompt versioning in production: git, registries, labels and rollbacks","How to version prompts in production: what belongs in a version, git versus a prompt registry, labels and rollbacks, eval gates in CI and cache side effects.","Five upright cards in a row, the third turned back on a curved arrow.",{"slug":722,"published":5,"minutes":723,"category":7,"tags":724,"keywords":728,"about":739,"sources":747,"cover":768,"og":769,"expertise":63,"locales":770,"lang":65,"title":771,"description":772,"coverAlt":773},"llm-batch-api-offline-workloads",9,[725,12,11,726,727],"Batch API","Prompt caching","Data pipelines",[729,730,731,732,733,734,735,736,737,738],"llm batch api","anthropic message batches api","openai batch api","gemini batch mode","batch inference llm cost","batch api 50 percent discount","openai flex processing","bedrock batch inference","llm offline processing","batch api prompt caching",[740,743,746],{"name":741,"url":742},"Batch processing","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FBatch_processing",{"name":744,"url":745},"JSON Lines","https:\u002F\u002Fjsonlines.org\u002F",{"name":31,"url":32},[748,751,753,754,757,760,762,765],{"title":749,"url":750},"Claude Platform Docs: Batch processing","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fbatch-processing",{"title":752,"url":630},"Claude Platform Docs: Rate limits (Message Batches API)",{"title":713,"url":714},{"title":755,"url":756},"OpenAI API Docs: Batch API","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fbatch",{"title":758,"url":759},"OpenAI API Docs: Flex processing","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fflex-processing",{"title":761,"url":636},"OpenAI API Docs: Rate limits (batch queue limits)",{"title":763,"url":764},"Google AI for Developers: Gemini Batch API","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fbatch-api",{"title":766,"url":767},"Amazon Bedrock User Guide: Batch inference","https:\u002F\u002Fdocs.aws.amazon.com\u002Fbedrock\u002Flatest\u002Fuserguide\u002Fbatch-inference.html","\u002Fimages\u002Fblog\u002Fllm-batch-api-offline-workloads\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-batch-api-offline-workloads\u002Fog.jpg",[65,66,67],"LLM batch APIs: half-price inference for work that can wait","When to use the Anthropic, OpenAI and Gemini batch APIs, how limits, result matching and failures work, and how batch discounts stack with prompt caching.","A wide box holding eight small squares, tiny squares arriving from the left, one long arrow returning.",1792245345455]