[{"data":1,"prerenderedAt":663},["ShallowReactive",2],{"blog-openrouter-en":3},{"slug":4,"published":5,"minutes":6,"category":7,"tags":8,"keywords":14,"about":22,"sources":26,"cover":44,"og":45,"expertise":46,"locales":47,"lang":48,"title":51,"description":52,"coverAlt":53,"metaTitle":54,"takeaways":55,"faq":61,"toc":74,"blocks":99,"others":424},"openrouter","2026-07-23",10,"llmops",[9,10,11,12,13],"LLM gateway","Model routing","Fallbacks","OpenAI-compatible","Pay per token",[4,15,16,17,18,19,20,21],"openrouter vs litellm","llm gateway comparison","openrouter pricing","openai compatible api gateway","llm fallback routing","multi model api gateway","byok llm routing",[23],{"name":24,"url":25},"OpenRouter","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FOpenRouter",[27,30,33,36,39,42],{"title":28,"url":29},"OpenRouter documentation: quickstart","https:\u002F\u002Fopenrouter.ai\u002Fdocs\u002Fquickstart",{"title":31,"url":32},"OpenRouter pricing","https:\u002F\u002Fopenrouter.ai\u002Fpricing",{"title":34,"url":35},"OpenRouter documentation: model fallbacks","https:\u002F\u002Fopenrouter.ai\u002Fdocs\u002Fguides\u002Frouting\u002Fmodel-fallbacks",{"title":37,"url":38},"OpenRouter documentation: provider routing","https:\u002F\u002Fopenrouter.ai\u002Fdocs\u002Fguides\u002Frouting\u002Fprovider-selection",{"title":40,"url":41},"OpenRouter documentation index","https:\u002F\u002Fopenrouter.ai\u002Fdocs\u002Fllms.txt",{"title":43,"url":25},"Wikipedia: OpenRouter","\u002Fimages\u002Fblog\u002Fopenrouter\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fopenrouter\u002Fog.jpg","ai-engineer",[48,49,50],"en","de","hu","OpenRouter: one API key in front of every model you might call","OpenRouter puts 500+ models from 80+ providers behind one OpenAI-compatible endpoint, with fallbacks and pass-through pricing. What it costs, where it breaks.","Nine small squares in a wide arc, each joined by a connector to one narrow strip.","OpenRouter review: one key, every model · Balázs Csorba",[56,57,58,59,60],"OpenRouter is a hosted routing layer: one OpenAI-compatible endpoint in front of 500+ models from 80+ providers, billed at the provider list price with a 5.5% fee on credit purchases.","Default routing is price-weighted: among providers with no outage in the last 30 seconds, the router picks by the inverse square of the price and keeps the rest as fallbacks.","Failed and fallback attempts are not billed for model tokens, so failover costs latency rather than tokens.","The fee is taken when credits are bought, never per request, which makes the overhead a flat percentage you can budget instead of a line item you have to audit.","Zero data retention is available on every plan, but keeping prompts inside the EU or the US starts at the Business plan, where the platform fee is 8%.",[62,65,68,71],{"q":63,"a":64},"Does OpenRouter mark up model prices?","The pricing page says no: inference is billed at the provider list price on every plan, and the platform fee of 5.5% on Standard and 8% on Business is charged when credits are purchased. Routing through openrouter\u002Fauto adds no fee either, so you pay the rate of whichever model serves the request.",{"q":66,"a":67},"How much latency does the gateway add?","It adds one network hop before the provider, and the documentation states plainly that routing improves reliability while latency varies by model, provider and region. Its advice for consistent latency is to pin a model and a provider, which switches load balancing off and makes the path behave like a direct API call.",{"q":69,"a":70},"What happens when a fallback triggers?","Any error can trigger the next model in the models array, including context-length validation, moderation flags, rate limits and downtime. The attempt that errors is not charged for its model tokens, so the bill covers the run that answers, but the caller waits for both.",{"q":72,"a":73},"Can prompts be kept inside one region?","Yes, on Business and Enterprise: requests to eu.openrouter.ai or us.openrouter.ai keep prompts and completions inside that region, with no cross-region fallback. Zero data retention is available on every plan, account-wide or per request.",[75,78,81,84,87,90,93,96],{"id":76,"title":77},"what-it-is","What it is",{"id":79,"title":80},"how-it-works","How it works",{"id":82,"title":83},"getting-started","Getting started",{"id":85,"title":86},"pricing","Pricing",{"id":88,"title":89},"control-plane","Control plane",{"id":91,"title":92},"where-it-shingles","Where it shingles",{"id":94,"title":95},"verdict","Verdict",{"id":97,"title":98},"sources","Sources",[100,104,107,110,113,151,152,162,168,172,175,178,216,217,220,222,229,244,245,248,265,271,272,275,285,291,292,298,354,357,374,375,378,393,403,404],{"type":101,"content":102},"paragraph",[103],"OpenRouter is a hosted routing layer for model APIs: one OpenAI-compatible endpoint that forwards each request to whichever provider is serving the chosen model. It hosts no models of its own, and the pricing page states that inference is billed at the provider list price, with the platform fee charged when credits are bought. This review takes the view that it is the cheapest way to put a changing catalogue behind one key, and the most expensive dependency to remove once every service speaks its dialect.",{"type":101,"content":105},[106],"It occupies the slot a self-hosted gateway would occupy, competing with LiteLLM, Portkey and the habit of calling each provider directly. The distinction is ownership rather than features. A gateway is a process you deploy, patch and scale; OpenRouter is a service in the request path of every call the product makes, so availability, data policy and pricing become somebody else’s problem, which is precisely what is being bought.",{"type":108,"level":109,"id":76,"text":77},"heading",2,{"type":101,"content":111},[112],"The company has routed traffic since 2023 and describes the product as a unified interface for hundreds of models. The documented surface is deliberately small: one endpoint, one key, a model slug and a few optional routing hints. Provider health, price comparison and failover all happen on the far side of that call.",{"type":114,"ordered":115,"items":116},"list",false,[117,124,126,128,134,147,149],[118,119,123],"Hosted only: there is no self-hosted edition, and the product is the endpoint at ",{"tag":120,"children":121},"code",[122],"https:\u002F\u002Fopenrouter.ai\u002Fapi\u002Fv1",".",[125],"500+ models from 80+ providers on the paid plans; the Free plan exposes 25+ free models across 4 providers.",[127],"OpenAI-compatible chat completions, plus an Anthropic messages route, a Responses API, embeddings and batches.",[129,130,133],"Model fallbacks through a ",{"tag":120,"children":131},[132],"models"," array: the next model is tried when the first errors, rate-limits or is refused by moderation.",[135,136,139,140,139,143,146],"Provider control per request — ",{"tag":120,"children":137},[138],"order",", ",{"tag":120,"children":141},[142],"only",{"tag":120,"children":144},[145],"ignore",", quantisation filters, and sorting by price, throughput or latency.",[148],"Bring-your-own-key routing, with the first $25,000 of list-price inference per month on a provider key charged at no OpenRouter fee.",[150],"Activity logs with export on every plan, per-generation records, and trace export to Datadog, Grafana Cloud, Langfuse, OpenTelemetry, Snowflake or a webhook.",{"type":108,"level":109,"id":79,"text":80},{"type":101,"content":153},[154,155,158,159,161],"The default strategy is price-based load balancing. The router first drops providers with a significant outage in the last 30 seconds, then picks among the stable ones weighted by the inverse square of the price, keeping the remainder as fallbacks: between a $1 endpoint and a $3 endpoint the cheap one is nine times more likely to be tried first. Any explicit ",{"tag":120,"children":156},[157],"sort"," or ",{"tag":120,"children":160},[138]," disables that balancing, which is the switch that turns a cost-optimising router into a predictable one.",{"type":163,"src":164,"alt":165,"wide":166,"caption":167},"image","\u002Fimages\u002Fblog\u002Fopenrouter\u002Fdiagram-7.webp","OpenRouter request path An application sends a chat completion to one OpenAI-compatible endpoint. The router sorts candidate providers by price, the fallback list names the next model, a provider answers, and the response reports the model that actually served. Workspaces, guardrails and activity logs sit underneath the request path.",true,"One endpoint replaces a rack of provider integrations. Routing, failover and billing all live on the far side of the call, which is exactly what the platform fee buys.",{"type":108,"level":169,"id":170,"text":171},3,"latency-cost","The cost of the hop",{"type":101,"content":173},[174],"Every call now crosses two networks instead of one. The pricing FAQ states plainly that routing improves reliability while latency varies by model, provider and region, and its advice for consistent latency is to pin a model and a provider. The extra hop is negligible next to a generation that runs for seconds and is not negligible next to a sub-second classification call: pinning returns the behaviour of a direct API plus one round trip.",{"type":101,"content":176},[177],"The indirection stays visible where it matters. The response names the endpoint that actually served the request, and an opt-in header surfaces the routing decision on every response, so an unexpected provider shows up in a log line rather than in a customer complaint.",{"type":114,"ordered":115,"items":179},[180,194,202,207],[181,183,184,139,187,158,190,193],{"tag":120,"children":182},[157],": ",{"tag":120,"children":185},[186],"price",{"tag":120,"children":188},[189],"throughput",{"tag":120,"children":191},[192],"latency",", and it switches load balancing off.",[195,197,198,201],{"tag":120,"children":196},[138],": an explicit provider sequence, with ",{"tag":120,"children":199},[200],"allow_fallbacks"," false when nothing outside it may be tried.",[203,206],{"tag":120,"children":204},[205],"require_parameters",": only providers that support every parameter in the request, which stops structured outputs and tool calls from being quietly degraded.",[208,211,212,215],{"tag":120,"children":209},[210],"zdr"," and ",{"tag":120,"children":213},[214],"data_collection",": route only to zero-data-retention endpoints, or only to providers that do not store inputs.",{"type":108,"level":109,"id":82,"text":83},{"type":101,"content":218},[219],"One key and one POST. Free accounts get a working endpoint immediately: free models are capped at 20 requests per minute and 50 per day, and the daily allowance rises to 1,000 once the account has bought at least $10 of credits.",{"type":120,"code":221},"curl https:\u002F\u002Fopenrouter.ai\u002Fapi\u002Fv1\u002Fchat\u002Fcompletions \\\n  -H \"Authorization: Bearer $OPENROUTER_API_KEY\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"models\": [\"~openai\u002Fgpt-sol-latest\", \"~anthropic\u002Fclaude-sonnet-latest\"],\n    \"messages\": [\n      { \"role\": \"user\", \"content\": \"Summarise this stack trace in one sentence.\" }\n    ],\n    \"provider\": {\n      \"sort\": \"price\",\n      \"require_parameters\": true\n    }\n  }'",{"type":101,"content":223},[224,225,228],"The answer comes back in the OpenAI shape, and the ",{"tag":120,"children":226},[227],"model"," field names the endpoint that actually served it. That is also the model the request is billed at, whether or not it was first in the list. An existing OpenAI client needs only a different base URL; the documentation shows the same swap in Python and TypeScript.",{"type":230,"variant":231,"title":232,"body":233},"callout","tip","Aliases versus pins",[234],[235,236,239,240,243],"Slugs beginning with ",{"tag":120,"children":237},[238],"~"," resolve to the newest model in a family, and the documentation’s own examples use ",{"tag":120,"children":241},[242],"~openai\u002Fgpt-sol-latest"," so deployments pick up releases without a redeploy. The mechanism cuts both ways: when an invoice, a benchmark or a reproduction has to name an exact model, use the full identifier and expect a 404 once it is retired.",{"type":108,"level":109,"id":85,"text":86},{"type":101,"content":246},[247],"There is no subscription, no minimum and no lock-in. Credits are bought upfront by card, AliPay or USDC, and each request deducts the provider list price from the balance; the platform fee, 5.5% on Standard and 8% on Business, is charged on the purchase and never on the request.",{"type":114,"ordered":115,"items":249},[250,252,254,263],[251],"Free: 25+ free models, 4 providers, activity logs with export, 50 requests per day on free models. No auto-routing, no budgets, no prompt caching.",[253],"Standard: the full 500+ models and 80+ providers at a 5.5% fee on credit purchases, with auto-routing, budgets, prompt caching, management API keys and five workspaces.",[255,256,211,259,262],"Business: the same catalogue at 8%, EU or US in-region routing through ",{"tag":120,"children":257},[258],"eu.openrouter.ai",{"tag":120,"children":260},[261],"us.openrouter.ai",", 1,000 workspaces and workload identity federation.",[264],"Enterprise: contracted fees, invoicing, SSO and SCIM, contractual SLAs, and $200,000 of BYOK list-price inference per month before a 5% charge applies.",{"type":230,"variant":266,"title":267,"body":268},"note","Where the percentage lands",[269],[270],"Because the fee is fixed at the purchase, overhead on inference is a flat percentage regardless of which model answers: $10,000 of list-price traffic on Standard costs $550 a month. Paid plans also buy routing features rather than throughput — OpenRouter sets no plan-based rate limit, so a limit hit on a paid model normally comes from the provider.",{"type":108,"level":109,"id":88,"text":89},{"type":101,"content":273},[274],"For teams, the parts that matter are not in the request body. Workspaces, budgets and guardrails decide who may call what with which key, and they live in the platform rather than in your code, which is the point: policy changes stop being deployments.",{"type":114,"ordered":115,"items":276},[277,279,281,283],[278],"Workspaces separate environments: five on Free and Standard, 1,000 on Business, each with its own keys and limits, plus a credit cap per key.",[280],"Guardrails attach to keys or members and cover spend, model access, prompt-injection patterns and sensitive-information rules.",[282],"Management API keys create, list and delete keys programmatically, so key lifecycle belongs in CI rather than in a dashboard habit.",[284],"Broadcast ships traces to Datadog, Grafana Cloud, Langfuse, OpenTelemetry, Sentry, Snowflake or any HTTP endpoint, which is what makes spend joinable with the rest of the stack.",{"type":101,"content":286},[287,288,290],"Zero data retention runs on every plan, account-wide or per request with ",{"tag":120,"children":289},[210],". Regional residency is the feature that is not free: keeping prompts and completions inside the EU or the US starts at Business, which makes the 8% plan the baseline rather than the upgrade for regulated workloads.",{"type":108,"level":109,"id":91,"text":92},{"type":101,"content":293},[294,295,297],"The weaknesses are structural. Every request depends on a third party’s availability, and the routing decision stays invisible unless the metadata header is enabled. Parameter support differs between providers of the same model, so a call that works through one endpoint can be quietly degraded through another unless ",{"tag":120,"children":296},[205]," is set. Provider price changes flow straight through — the FAQ states that you will be charged at the new rate — and the free tier is rate-limited rather than generous. There is also ownership: Bloomberg and the Wall Street Journal reported in August 2026 that Stripe had agreed to buy the company, a useful reminder that the layer inside the request path has an owner.",{"type":299,"head":300,"rows":308},"table",[301,303,304,306],[302],"",[24],[305],"LiteLLM",[307],"First-party APIs",[309,318,327,336,345],[310,312,314,316],[311],"Deployment",[313],"Hosted, no self-hosted edition",[315],"Self-hosted proxy you operate",[317],"Your code per provider",[319,321,323,325],[320],"Fee",[322],"5.5% on credits (Standard)",[324],"Free under MIT, paid tiers by quote",[326],"None",[328,330,332,334],[329],"Catalogue",[331],"500+ models, 80+ providers",[333],"140+ providers, ~1,900 models",[335],"One vendor per integration",[337,339,341,343],[338],"Fallback",[340],"Per request, cross-provider and cross-model",[342],"Configured in the router",[344],"Written by hand",[346,348,350,352],[347],"Best fit",[349],"Many models, one bill, no operations",[351],"Platform team owning keys and budgets",[353],"One model on a hot path",{"type":101,"content":355},[356],"The comparison that matters is custody rather than features. OpenRouter removes the operations and adds a percentage plus a dependency; LiteLLM keeps the keys and the workload inside your infrastructure; first-party APIs give the shortest path and the least cover when a model is unavailable. For a product that changes models as often as others change a pricing page, that cover is worth more than the fee.",{"type":230,"variant":358,"title":359,"body":360},"warn","Pin it before it pins you",[361],[362,363,158,365,367,368,370,371,373],"Default load balancing can move the next request to a different provider, which changes latency, quantisation and sometimes output. Set ",{"tag":120,"children":364},[157],{"tag":120,"children":366},[138]," explicitly, keep ",{"tag":120,"children":369},[205]," on for structured outputs, and decide on purpose whether ",{"tag":120,"children":372},[200]," may change the model family mid-request — by default an error moves the call to the next model in the list.",{"type":108,"level":109,"id":94,"text":95},{"type":101,"content":376},[377],"OpenRouter answers how to give many agents, services and teams access to a fast-changing model list without operating a gateway. It does not answer how to own the request path.",{"type":114,"ordered":166,"items":379},[380,385,387,389,391],[381,382,384],"Adopt it when the model list changes faster than the integration: one endpoint and a ",{"tag":120,"children":383},[132]," array turn a vendor evaluation into a configuration value.",[386],"Adopt it for agent and coding-agent workloads, where the bill spans many models and one activity log with export beats four provider dashboards.",[388],"Use BYOK when contracts or data policy require a direct relationship with the provider; routing and analytics are kept, and the first $25,000 of list-price inference per month is fee-free.",[390],"Skip it on latency-critical paths that cannot pin a provider: availability routing is the product, and a request that may change endpoint is a tail latency nobody controls.",[392],"Skip it when one provider and one model is the whole product: a first-party SDK, direct billing and no intermediary percentage is less machinery for the same call.",{"type":394,"content":395},"quote",[396,397,398,402],"“Inference is billed at the provider’s list price on every plan.”"," ",{"tag":399,"href":32,"children":400},"a",[401],"the pricing page"," The fee is small and legible; the dependency is the part that has to be priced in, and it is not listed anywhere.",{"type":108,"level":109,"id":97,"text":98},{"type":114,"ordered":166,"items":405},[406,409,412,415,418,421],[407],{"tag":399,"href":29,"children":408},[28],[410],{"tag":399,"href":32,"children":411},[31],[413],{"tag":399,"href":35,"children":414},[34],[416],{"tag":399,"href":38,"children":417},[37],[419],{"tag":399,"href":41,"children":420},[40],[422],{"tag":399,"href":25,"children":423},[43],[425,482,546,607],{"slug":426,"published":427,"minutes":428,"category":7,"tags":429,"keywords":435,"about":449,"sources":457,"cover":476,"og":477,"expertise":46,"locales":478,"lang":48,"title":479,"description":480,"coverAlt":481},"deepseek-v4-1-flash-1017","2026-10-17",15,[430,431,432,433,434],"DeepSeek","Open-weight models","KV cache","Mixture of experts","LLM cost",[436,437,438,439,440,441,442,443,444,445,446,447,448],"DeepSeek V4.1 Flash 1017","DeepSeek V4.1 Flash","Causal Encoder-Decoder","Compressed Sparse Attention 2","CSA2","FP4 KV cache","KV cache compression","SWA Bounded Replay","Engram conditional memory","DSpark speculative decoding","DeepSeek pricing","DeepSWE benchmark","open-weight coding model",[450,452,454],{"name":430,"url":451},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FDeepSeek",{"name":433,"url":453},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FMixture_of_experts",{"name":455,"url":456},"Transformer (deep learning architecture)","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FTransformer_(deep_learning_architecture)",[458,461,464,467,470,473],{"title":459,"url":460},"DeepSeek-AI: DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression (arXiv:2609.19969)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.19969",{"title":462,"url":463},"Hugging Face: deepseek-ai\u002FDeepSeek-V4.1-Flash-1017 model card","https:\u002F\u002Fhuggingface.co\u002Fdeepseek-ai\u002FDeepSeek-V4.1-Flash-1017",{"title":465,"url":466},"OpenRouter: DeepSeek V4.1 Flash 1017, pricing and providers","https:\u002F\u002Fopenrouter.ai\u002Fdeepseek\u002Fdeepseek-v4.1-flash-1017",{"title":468,"url":469},"The Decoder: Deepseek V4.1 Flash 1017 beats larger rivals in agentic tasks while once again massively cutting per-task costs","https:\u002F\u002Fthe-decoder.com\u002Fdeepseek-v4-1-flash-1017-beats-larger-rivals-in-agentic-tasks-while-once-again-massively-cutting-per-task-costs\u002F",{"title":471,"url":472},"AI Weekly: DeepSeek opens V4.1-Flash-1017 weights and $0.75 Search API","https:\u002F\u002Faiweekly.co\u002Falerts\u002Fdeepseek-opens-v41-flash-1017-weights-and-075-search-api",{"title":474,"url":475},"DeepSeek on X: DeepSeek-V4.1-Flash-1017 announcement","https:\u002F\u002Fx.com\u002Fdeepseek_ai\u002Fstatus\u002F2111375640717033777","\u002Fimages\u002Fblog\u002Fdeepseek-v4-1-flash-1017\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fdeepseek-v4-1-flash-1017\u002Fog.jpg",[48,49,50],"DeepSeek V4.1 Flash 1017: 890 bytes of KV cache per token, and what that buys an agent","A technical deep dive into DeepSeek V4.1 Flash 1017: the encoder-decoder split, CSA2 cache sharing, FP4 KV cache, the benchmarks and the real cost per agent task.","A tall stack of thin cards on the left, an arrow, and the same cards pressed into a stack a quarter of the height on the right.",{"slug":483,"published":427,"minutes":6,"category":7,"tags":484,"keywords":488,"about":500,"sources":510,"cover":540,"og":541,"expertise":46,"locales":542,"lang":48,"title":543,"description":544,"coverAlt":545},"llm-rate-limits-retries-provider-fallback",[485,486,338,487,9],"Rate limits","Retries","Reliability",[489,490,491,492,493,494,495,496,497,498,499],"llm rate limit handling","429 too many requests llm api","exponential backoff with jitter","anthropic 529 overloaded error","openai rate limit retry","llm provider fallback","litellm fallbacks","retry-after header","llm circuit breaker","multi provider llm failover","claude api rate limits",[501,504,507],{"name":502,"url":503},"Token bucket","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FToken_bucket",{"name":505,"url":506},"Exponential backoff","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FExponential_backoff",{"name":508,"url":509},"Circuit breaker design pattern","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FCircuit_breaker_design_pattern",[511,514,517,520,523,526,529,532,535,537],{"title":512,"url":513},"Claude Platform Docs: Rate limits","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fapi\u002Frate-limits",{"title":515,"url":516},"Claude Platform Docs: API errors","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fapi\u002Ferrors",{"title":518,"url":519},"OpenAI API Docs: Rate limits","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Frate-limits",{"title":521,"url":522},"AWS Builders Library: Timeouts, retries and backoff with jitter","https:\u002F\u002Fbuilder.aws.com\u002Fcontent\u002F3EumjoZascWd1oZiEgL8ORlv3qE\u002Ftimeouts-retries-and-backoff-with-jitter",{"title":524,"url":525},"MDN: Retry-After header","https:\u002F\u002Fdeveloper.mozilla.org\u002Fen-US\u002Fdocs\u002FWeb\u002FHTTP\u002FReference\u002FHeaders\u002FRetry-After",{"title":527,"url":528},"Martin Fowler: Circuit Breaker (2014)","https:\u002F\u002Fmartinfowler.com\u002Fbliki\u002FCircuitBreaker.html",{"title":530,"url":531},"LiteLLM Docs: Router, retries and cooldowns","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Frouting",{"title":533,"url":534},"LiteLLM Docs: Proxy fallbacks and reliability","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Fproxy\u002Freliability",{"title":536,"url":35},"OpenRouter Docs: Model fallbacks",{"title":538,"url":539},"AI SDK Docs: Settings (maxRetries, timeout)","https:\u002F\u002Fai-sdk.dev\u002Fdocs\u002Fai-sdk-core\u002Fsettings","\u002Fimages\u002Fblog\u002Fllm-rate-limits-retries-provider-fallback\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-rate-limits-retries-provider-fallback\u002Fog.jpg",[48,49,50],"LLM rate limits in production: retries, backoff and provider fallback","How LLM rate limits work, which errors to retry, backoff with jitter, retrying in one layer only, and fallback chains that do not break your output.","Small squares climbing three steps up to a wide rectangle, two more rectangles behind it in a row.",{"slug":547,"published":427,"minutes":548,"category":7,"tags":549,"keywords":555,"about":566,"sources":576,"cover":601,"og":602,"expertise":46,"locales":603,"lang":48,"title":604,"description":605,"coverAlt":606},"prompt-versioning-management",8,[550,551,552,553,554],"Prompt management","Versioning","LLMOps","Evals","Langfuse",[556,557,558,559,560,561,562,563,564,565],"prompt versioning","prompt management","prompt registry","version control for prompts","langfuse prompt management","mlflow prompt registry","prompts in git","openai prompt object deprecation","prompt rollback production","prompt ci evaluation",[567,570,573],{"name":568,"url":569},"Prompt engineering","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FPrompt_engineering",{"name":571,"url":572},"Version control","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FVersion_control",{"name":574,"url":575},"Continuous integration","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FContinuous_integration",[577,580,583,586,589,592,595,598],{"title":578,"url":579},"OpenAI API Docs: Migrate from prompt objects","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fprompting\u002Fmigrate-from-prompt-object",{"title":581,"url":582},"OpenAI API Docs: Prompt engineering (reusable prompts deprecation note)","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fprompt-engineering",{"title":584,"url":585},"Langfuse Docs: Prompt version control","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Fprompt-version-control",{"title":587,"url":588},"Langfuse Docs: Prompt caching in the SDKs","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Fcaching",{"title":590,"url":591},"Langfuse Docs: Link prompts to traces","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Flink-to-traces",{"title":593,"url":594},"MLflow Docs: Prompt Registry","https:\u002F\u002Fmlflow.org\u002Fdocs\u002Flatest\u002Fgenai\u002Fprompt-registry\u002F",{"title":596,"url":597},"Promptfoo Docs: GitHub Action","https:\u002F\u002Fwww.promptfoo.dev\u002Fdocs\u002Fintegrations\u002Fgithub-action\u002F",{"title":599,"url":600},"Claude Platform Docs: Prompt caching","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fprompt-caching","\u002Fimages\u002Fblog\u002Fprompt-versioning-management\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fprompt-versioning-management\u002Fog.jpg",[48,49,50],"Prompt versioning in production: git, registries, labels and rollbacks","How to version prompts in production: what belongs in a version, git versus a prompt registry, labels and rollbacks, eval gates in CI and cache side effects.","Five upright cards in a row, the third turned back on a curved arrow.",{"slug":608,"published":427,"minutes":609,"category":7,"tags":610,"keywords":615,"about":626,"sources":636,"cover":657,"og":658,"expertise":46,"locales":659,"lang":48,"title":660,"description":661,"coverAlt":662},"llm-batch-api-offline-workloads",9,[611,612,552,613,614],"Batch API","Cost","Prompt caching","Data pipelines",[616,617,618,619,620,621,622,623,624,625],"llm batch api","anthropic message batches api","openai batch api","gemini batch mode","batch inference llm cost","batch api 50 percent discount","openai flex processing","bedrock batch inference","llm offline processing","batch api prompt caching",[627,630,633],{"name":628,"url":629},"Batch processing","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FBatch_processing",{"name":631,"url":632},"JSON Lines","https:\u002F\u002Fjsonlines.org\u002F",{"name":634,"url":635},"Large language model","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FLarge_language_model",[637,640,642,643,646,649,651,654],{"title":638,"url":639},"Claude Platform Docs: Batch processing","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fbatch-processing",{"title":641,"url":513},"Claude Platform Docs: Rate limits (Message Batches API)",{"title":599,"url":600},{"title":644,"url":645},"OpenAI API Docs: Batch API","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fbatch",{"title":647,"url":648},"OpenAI API Docs: Flex processing","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fflex-processing",{"title":650,"url":519},"OpenAI API Docs: Rate limits (batch queue limits)",{"title":652,"url":653},"Google AI for Developers: Gemini Batch API","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fbatch-api",{"title":655,"url":656},"Amazon Bedrock User Guide: Batch inference","https:\u002F\u002Fdocs.aws.amazon.com\u002Fbedrock\u002Flatest\u002Fuserguide\u002Fbatch-inference.html","\u002Fimages\u002Fblog\u002Fllm-batch-api-offline-workloads\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-batch-api-offline-workloads\u002Fog.jpg",[48,49,50],"LLM batch APIs: half-price inference for work that can wait","When to use the Anthropic, OpenAI and Gemini batch APIs, how limits, result matching and failures work, and how batch discounts stack with prompt caching.","A wide box holding eight small squares, tiny squares arriving from the left, one long arrow returning.",1792261156195]