[{"data":1,"prerenderedAt":565},["ShallowReactive",2],{"blog-litellm-en":3},{"slug":4,"published":5,"minutes":6,"category":7,"tags":8,"keywords":14,"about":22,"sources":29,"cover":51,"og":52,"expertise":53,"locales":54,"lang":55,"title":58,"description":59,"coverAlt":60,"metaTitle":61,"takeaways":62,"faq":68,"toc":81,"blocks":106,"others":325},"litellm","2026-05-20",10,"llmops",[9,10,11,12,13],"LLM gateway","OpenAI-compatible","Cost control","Routing","MCP",[4,15,16,17,18,19,20,21],"litellm vs portkey","llm gateway comparison","openai compatible proxy","litellm proxy config yaml","llm spend tracking per team","self-hosted llm gateway","llm budget enforcement",[23,26],{"name":24,"url":25},"LiteLLM","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FLiteLLM",{"name":27,"url":28},"Model Context Protocol","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FModel_Context_Protocol",[30,33,36,39,42,45,48],{"title":31,"url":32},"LiteLLM documentation: getting started","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002F",{"title":34,"url":35},"LiteLLM documentation: virtual keys","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Fproxy\u002Fvirtual_keys",{"title":37,"url":38},"LiteLLM documentation: router and load balancing","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Frouting",{"title":40,"url":41},"LiteLLM documentation: benchmarks","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Fbenchmarks",{"title":43,"url":44},"LiteLLM documentation: supported endpoints","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Fsupported_endpoints",{"title":46,"url":47},"LiteLLM documentation: Enterprise","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Fenterprise",{"title":49,"url":50},"LiteLLM pricing","https:\u002F\u002Fwww.litellm.ai\u002Fpricing","\u002Fimages\u002Fblog\u002Flitellm\u002Fcover.webp","\u002Fimages\u002Fblog\u002Flitellm\u002Fog.jpg","ai-engineer",[55,56,57],"en","de","hu","LiteLLM: the OpenAI-compatible gateway most platform teams end up running","LiteLLM is the MIT-licensed OpenAI-compatible gateway most platform teams put in front of their providers. What it does well, what it costs and where it breaks.","Five modules in a half-circle sending arrows into one tall slab, and one arrow continuing right.","LiteLLM review: the gateway in the middle · Balázs Csorba",[63,64,65,66,67],"LiteLLM is an MIT-licensed Python SDK and proxy that puts 140+ providers behind one OpenAI-shaped endpoint, with virtual keys, per-team budgets and cross-provider fallbacks.","The value is organisational rather than technical: changing model becomes a config edit, and every request produces a spend row.","Overhead is measurable and published: the docs report 8ms p95 at 1,000 requests per second, and are unusually explicit that latency only means something next to the number of requests in flight.","The free tier is the whole gateway. SSO beyond five users, audit logs, key rotation and the built-in moderation callbacks sit behind a licence quoted by annual request capacity, with no public list price.","Release cadence is roughly one minor line a week against a four-line support window, so the upgrade is a recurring operational task rather than an event.",[69,72,75,78],{"q":70,"a":71},"Is LiteLLM just a wrapper around the OpenAI SDK?","No. The SDK normalises requests, responses and exceptions across the supported providers, but the proxy adds virtual keys, spend tracking, budgets, routing, cooldowns, fallbacks and caching on top. The usual path is to start with the SDK inside one service and move to the proxy once more than one team needs access.",{"q":73,"a":74},"How much latency does the gateway add?","The benchmark page reports 8ms p95 at 1,000 requests per second on four instances of 4 vCPU and 8 GB. That figure includes the client: the docs point out that with no think time a closed-loop client holds 1,000 requests in flight instead of about 130, which by Little's Law reports roughly eight times the latency at the same throughput.",{"q":76,"a":77},"Does the free version really enforce budgets?","Yes. Virtual keys, teams, spend tracking, budgets, rate limits and LLM fallbacks are all in the open-source tier. What the licence gates is identity and governance: SSO beyond five users, SCIM, audit logs, secret-manager write-back, virtual key rotation and the built-in moderation callbacks.",{"q":79,"a":80},"How is Enterprise priced?","There is no public list price. The pricing page says the licence is sized to annual gateway request capacity, deployment architecture and support needs, never per token, and every quote comes from sales. A 30-day trial key is issued without a credit card, and procurement is also available through AWS Marketplace and resellers.",[82,85,88,91,94,97,100,103],{"id":83,"title":84},"what-it-is","What LiteLLM is",{"id":86,"title":87},"how-it-works","How it works",{"id":89,"title":90},"getting-started","Getting started",{"id":92,"title":93},"keys-and-budgets","Keys, budgets and spend",{"id":95,"title":96},"pricing","Pricing and licence boundaries",{"id":98,"title":99},"where-it-shingles","Where it shingles",{"id":101,"title":102},"verdict","Verdict",{"id":104,"title":105},"sources","Sources",[107,111,114,117,120,138,139,142,148,151,155,158,161,173,174,177,179,186,208,209,212,214,217,218,221,224,225,228,275,278,279,282,294,300,301],{"type":108,"content":109},"paragraph",[110],"LiteLLM is the open-source LLM gateway that most platform teams end up running whether or not they meant to. It is two products in one package: a Python SDK that normalises the major providers onto the OpenAI request and response shape, and a proxy server that puts virtual keys, budgets, routing and spend tracking in front of them. On the documentation and the published numbers it is the most complete piece of plumbing in this category, and heavy enough that it deserves a deliberate decision rather than an accidental one.",{"type":108,"content":112},[113],"It sits one hop in front of the model providers, not in front of the application data. Retrieval, prompt management and agent orchestration live elsewhere; an OpenAI client, a LangGraph workflow and a LiteLLM proxy coexist without complaint. What the gateway owns is the request itself: who is allowed to make it, which deployment serves it, what it costs and who is accountable for the bill.",{"type":115,"level":116,"id":83,"text":84},"heading",2,{"type":108,"content":118},[119],"The name covers two runnable things that share a config file and a model price table. The SDK is a library you import. The proxy is a server that speaks the OpenAI wire format, so any client that already works against OpenAI can be redirected by changing its base URL and its key.",{"type":121,"ordered":122,"items":123},"list",false,[124,126,128,130,132,134,136],[125],"MIT-licensed, with an Enterprise tier that adds identity and governance features rather than capacity.",[127],"140+ provider integrations and roughly 1,900 models, tracked in a public price and context-window file that ships with the package.",[129],"OpenAI-compatible endpoints including chat completions, the Anthropic messages route, the responses API, embeddings, batches and realtime.",[131],"Virtual keys with per-key, per-team and per-user budgets, rate limits and model allowlists, all backed by Postgres.",[133],"A router with several deployment-selection strategies, per-deployment cooldowns, weighted failover, cross-model fallbacks and session affinity.",[135],"An MCP gateway and an A2A agent gateway on the same endpoint and the same key policy, so agents get governed tool access rather than a separate one.",[137],"Callbacks into Langfuse, MLflow, Helicone, LangSmith and OpenTelemetry, plus Prometheus metrics and an admin UI.",{"type":115,"level":116,"id":86,"text":87},{"type":108,"content":140},[141],"Every request through the proxy follows the same order of operations: authenticate the key, check the budget and rate limits, count tokens, pick a deployment, call the provider, translate the response, record the cost. Each step can fail on its own, and the router's behaviour on failure is where most of the operational learning happens.",{"type":143,"src":144,"alt":145,"wide":146,"caption":147},"image","\u002Fimages\u002Fblog\u002Flitellm\u002Fdiagram-7.webp","LiteLLM request path An application sends a chat completion request to the LiteLLM gateway. The gateway checks the virtual key and the budget, the router picks a deployment, the provider answers, and the same OpenAI-shaped JSON goes back. Postgres, Redis and observability callbacks sit underneath the request path.",true,"One hop in front of every provider. Postgres and Redis are what turn a request translator into a control point, and they are also what has to be sized and backed up.",{"type":108,"content":149},[150],"Model names are the abstraction that makes this work. The config file lists deployments under a shared alias, several deployments may carry the same alias, and a request for that alias is answered by whichever deployment the routing strategy picks. Move the alias and every caller changes provider without a code change. The same table holds the fallbacks: a failed deployment is either re-picked from its peers or escalated to a named fallback model.",{"type":115,"level":152,"id":153,"text":154},3,"overhead","Measured overhead",{"type":108,"content":156},[157],"The benchmark page reports 8ms p95 at 1,000 requests per second across four instances of 4 vCPU and 8 GB, and for the realtime endpoint 59ms median, 67ms p95 and 99ms p99 at 1,207 RPS. The same page publishes a head-to-head against Portkey at roughly 1,170 RPS on the same hardware, where LiteLLM reports p95 150ms and p99 240ms against Portkey's 230ms and 500ms.",{"type":108,"content":159},[160],"The more useful part is the caveat the docs volunteer. The load generator sleeps 0.5 to 1 second between requests, so about 130 requests are in flight at any instant; a closed-loop client with no think time holds 1,000 and therefore reports roughly eight times the latency at the same throughput. A published overhead number is meaningless without the in-flight depth next to it, which is more disclosure than most vendor benchmarks carry.",{"type":162,"variant":163,"title":164,"body":165},"callout","note","Measure it yourself",[166],[167,168,172],"The docs ship a ",{"tag":169,"children":170},"code",[171],"network_mock"," mode that intercepts outbound requests at the HTTP transport layer and returns canned responses, plus a public benchmark script. Running it against your own traffic takes minutes and produces the only overhead figure worth planning around.",{"type":115,"level":116,"id":89,"text":90},{"type":108,"content":175},[176],"The smallest useful deployment is a config file with two deployments behind one alias, a master key and a Postgres connection. Caching, guardrails, callbacks and the admin UI all come later.",{"type":169,"code":178},"# config.yaml: two deployments behind one alias, plus a managed fallback\nmodel_list:\n  - model_name: chat-fast\n    litellm_params:\n      model: openai\u002Fgpt-5.6-luna\n      api_key: os.environ\u002FOPENAI_API_KEY\n      rpm: 900\n  - model_name: chat-fast\n    litellm_params:\n      model: anthropic\u002Fclaude-sonnet-5\n      api_key: os.environ\u002FANTHROPIC_API_KEY\n      rpm: 60\n  - model_name: chat-cheap\n    litellm_params:\n      model: openai\u002Fgpt-5.6-luna\n      api_key: os.environ\u002FOPENAI_API_KEY\n\nrouter_settings:\n  routing_strategy: simple-shuffle\n  num_retries: 2\n  fallbacks:\n    - chat-fast: [chat-cheap]\n\ngeneral_settings:\n  master_key: os.environ\u002FLITELLM_MASTER_KEY\n  database_url: os.environ\u002FDATABASE_URL\n\n# docker run -p 4000:4000 -v $(pwd)\u002Fconfig.yaml:\u002Fapp\u002Fconfig.yaml \\\n#   -e DATABASE_URL -e LITELLM_MASTER_KEY \\\n#   docker.litellm.ai\u002Fberriai\u002Flitellm:latest --config \u002Fapp\u002Fconfig.yaml",{"type":108,"content":180},[181,182,185],"The client change is two lines: point the OpenAI SDK at the proxy and pass a virtual key instead of the provider key. Every response carries an ",{"tag":169,"children":183},[184],"x-litellm-overhead-duration-ms"," header reporting what the gateway itself cost, which is the number to alert on when latency regresses and the upstream did not change.",{"type":162,"variant":187,"title":188,"body":189},"warn","Retries compound, and concurrency caps now reject",[190],[191,192,195,196,199,200,203,204,207],"The docs are explicit that ",{"tag":169,"children":193},[194],"num_retries"," is LiteLLM's own loop and that the provider SDK's ",{"tag":169,"children":197},[198],"max_retries"," is pinned to zero on routed calls, so a deployment value cannot be applied twice. Separately, a deployment whose ",{"tag":169,"children":201},[202],"max_parallel_requests"," or inferred ",{"tag":169,"children":205},[206],"rpm"," cap is saturated now returns 429 immediately instead of queueing. Earlier versions waited for a free slot, so code that relied on the queue needs the cap raised rather than the timeout lengthened.",{"type":115,"level":116,"id":92,"text":93},{"type":108,"content":210},[211],"The key model is where LiteLLM earns its keep. A virtual key carries a model allowlist, a maximum budget with a reset window, TPM and RPM limits, a parallel request cap and an owner. Keys inherit from their owner, which is convenient and also the sharpest edge in the design: a key minted by an admin without a user id inherits nothing at all, and a key owned by a proxy admin can reach the admin management routes because management access follows the owner's role rather than the key's own permissions.",{"type":169,"code":213},"# mint a key for one team, capped, rate-limited and attributed\ncurl 'http:\u002F\u002Flocalhost:4000\u002Fkey\u002Fgenerate' \\\n  -H \"Authorization: Bearer $LITELLM_MASTER_KEY\" \\\n  -H 'Content-Type: application\u002Fjson' \\\n  -d '{\n    \"models\": [\"chat-fast\"],\n    \"max_budget\": 40,\n    \"budget_duration\": \"30d\",\n    \"rpm_limit\": 600,\n    \"tpm_limit\": 400000,\n    \"metadata\": {\"team\": \"platform\", \"env\": \"prod\"}\n  }'\n\n# what has this key spent, and against which cap\ncurl 'http:\u002F\u002Flocalhost:4000\u002Fkey\u002Finfo?key=sk-...' \\\n  -H \"Authorization: Bearer $LITELLM_MASTER_KEY\"",{"type":108,"content":215},[216],"Spend is computed from the same public price file the SDK ships, written on every completion, embedding and image call, and queryable per key, user and team. Those numbers are therefore estimates derived from a price table that can lag a provider's own billing: close enough to charge a team back, not close enough to reconcile an invoice. When a cost figure matters for a contract, take it from the provider invoice.",{"type":115,"level":116,"id":95,"text":96},{"type":108,"content":219},[220],"The gateway is free to self-host forever under MIT, with no per-seat and no per-token charge. The pricing page lists the open-source tier as 140+ provider integrations, virtual keys, users and teams, spend tracking, budgets and rate limits, LLM fallbacks, request and response logging, Prometheus metrics and guardrails. Procurement runs directly or through AWS Marketplace and resellers.",{"type":108,"content":222},[223],"The licence buys control, not throughput. SSO is included up to five users and needs a licence beyond that; SCIM, audit logs, secret-manager write-back, IP allowlists, the multi-region control plane, virtual key rotation and the built-in moderation callbacks all sit behind it, and the docs name them individually so nobody discovers the boundary after rollout. Enterprise pricing is quoted against annual request capacity and deployment shape, never per token, and no list price is published, so any real cost comparison starts with a sales conversation.",{"type":115,"level":116,"id":98,"text":99},{"type":108,"content":226},[227],"The weaknesses are operational rather than functional. The proxy needs Postgres and, once it runs on more than one instance, Redis; that is a stateful system with schema migrations and connection arithmetic, and the docs carry dedicated sizing pages precisely because it is where deployments fail. The Python SDK pulls a large dependency tree into a service for what is mostly request formatting. And the March 2026 supply-chain incident, in which trojanised LiteLLM releases reached PyPI, is a reminder that this is a package with very high install counts handling production keys.",{"type":229,"head":230,"rows":238},"table",[231,233,234,236],[232],"",[24],[235],"Portkey",[237],"Bifrost",[239,248,257,266],[240,242,244,246],[241],"Deployment",[243],"Self-hosted, air-gapped supported",[245],"Managed cloud and self-hosted",[247],"Self-hosted",[249,251,253,255],[250],"Cost basis",[252],"Free OSS, licence by capacity",[254],"Per-month subscription",[256],"No published list price",[258,260,262,264],[259],"Vendor p99 overhead",[261],"0.66 ms",[263],"2.29 ms",[265],"4.54 ms",[267,269,271,273],[268],"Best fit",[270],"Platform team owning access",[272],"Guardrails first",[274],"Minimal hop overhead",{"type":108,"content":276},[277],"Those overhead figures come from LiteLLM's own benchmark, run with every gateway pointed at the same deterministic mock upstream on identical hardware, which makes the comparison fair and the winner inevitable. For anything that is not an LLM gateway, the Kubernetes-native option deserves a look: Envoy AI Gateway exposes the OpenAI routing surface without adding a Python runtime to the request path, which is the structural difference that matters more than any of the numbers above.",{"type":115,"level":116,"id":101,"text":102},{"type":108,"content":280},[281],"LiteLLM is the right default for a platform team that must give many teams access to several providers and needs three questions answered in production: who spent this, which deployment served it, and what happens when that provider degrades. It answers all three better than the alternatives, and it does so without taking custody of the provider keys.",{"type":121,"ordered":146,"items":283},[284,286,288,290,292],[285],"Adopt it when more than one team calls models through more than one provider. Budgets, spend attribution and cross-provider fallback are what justify the stateful deployment.",[287],"Adopt it when procurement consolidation is the goal. Moving from one model to another becomes a config edit rather than a security review cycle.",[289],"Take the SDK alone first if there is a single service. It normalises errors and responses for a fraction of the operational cost and leaves the door open to the proxy later.",[291],"Skip it if one team and one provider is the whole story. A provider SDK and a spend spreadsheet are less machinery for the same result.",[293],"Skip it if the last millisecond matters on the hot path. The gateway is a Python process in the request path, and the Rust rewrite is still labelled beta.",{"type":162,"variant":295,"title":296,"body":297},"tip","Before committing",[298],[299],"Pin a minor line, take its patches, and move up before it drops out of the four-line support window. Read the database sizing page before the first scale-out rather than after the connection errors start. And if the provider SDK is already installed everywhere, pin the exact version in a lockfile rather than trusting the resolver.",{"type":115,"level":116,"id":104,"text":105},{"type":121,"ordered":146,"items":302},[303,307,310,313,316,319,322],[304],{"tag":305,"href":32,"children":306},"a",[31],[308],{"tag":305,"href":35,"children":309},[34],[311],{"tag":305,"href":38,"children":312},[37],[314],{"tag":305,"href":41,"children":315},[40],[317],{"tag":305,"href":44,"children":318},[43],[320],{"tag":305,"href":47,"children":321},[46],[323],{"tag":305,"href":50,"children":324},[49],[326,383,448,509],{"slug":327,"published":328,"minutes":329,"category":7,"tags":330,"keywords":336,"about":350,"sources":358,"cover":377,"og":378,"expertise":53,"locales":379,"lang":55,"title":380,"description":381,"coverAlt":382},"deepseek-v4-1-flash-1017","2026-10-17",15,[331,332,333,334,335],"DeepSeek","Open-weight models","KV cache","Mixture of experts","LLM cost",[337,338,339,340,341,342,343,344,345,346,347,348,349],"DeepSeek V4.1 Flash 1017","DeepSeek V4.1 Flash","Causal Encoder-Decoder","Compressed Sparse Attention 2","CSA2","FP4 KV cache","KV cache compression","SWA Bounded Replay","Engram conditional memory","DSpark speculative decoding","DeepSeek pricing","DeepSWE benchmark","open-weight coding model",[351,353,355],{"name":331,"url":352},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FDeepSeek",{"name":334,"url":354},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FMixture_of_experts",{"name":356,"url":357},"Transformer (deep learning architecture)","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FTransformer_(deep_learning_architecture)",[359,362,365,368,371,374],{"title":360,"url":361},"DeepSeek-AI: DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression (arXiv:2609.19969)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.19969",{"title":363,"url":364},"Hugging Face: deepseek-ai\u002FDeepSeek-V4.1-Flash-1017 model card","https:\u002F\u002Fhuggingface.co\u002Fdeepseek-ai\u002FDeepSeek-V4.1-Flash-1017",{"title":366,"url":367},"OpenRouter: DeepSeek V4.1 Flash 1017, pricing and providers","https:\u002F\u002Fopenrouter.ai\u002Fdeepseek\u002Fdeepseek-v4.1-flash-1017",{"title":369,"url":370},"The Decoder: Deepseek V4.1 Flash 1017 beats larger rivals in agentic tasks while once again massively cutting per-task costs","https:\u002F\u002Fthe-decoder.com\u002Fdeepseek-v4-1-flash-1017-beats-larger-rivals-in-agentic-tasks-while-once-again-massively-cutting-per-task-costs\u002F",{"title":372,"url":373},"AI Weekly: DeepSeek opens V4.1-Flash-1017 weights and $0.75 Search API","https:\u002F\u002Faiweekly.co\u002Falerts\u002Fdeepseek-opens-v41-flash-1017-weights-and-075-search-api",{"title":375,"url":376},"DeepSeek on X: DeepSeek-V4.1-Flash-1017 announcement","https:\u002F\u002Fx.com\u002Fdeepseek_ai\u002Fstatus\u002F2111375640717033777","\u002Fimages\u002Fblog\u002Fdeepseek-v4-1-flash-1017\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fdeepseek-v4-1-flash-1017\u002Fog.jpg",[55,56,57],"DeepSeek V4.1 Flash 1017: 890 bytes of KV cache per token, and what that buys an agent","A technical deep dive into DeepSeek V4.1 Flash 1017: the encoder-decoder split, CSA2 cache sharing, FP4 KV cache, the benchmarks and the real cost per agent task.","A tall stack of thin cards on the left, an arrow, and the same cards pressed into a stack a quarter of the height on the right.",{"slug":384,"published":328,"minutes":6,"category":7,"tags":385,"keywords":390,"about":402,"sources":412,"cover":442,"og":443,"expertise":53,"locales":444,"lang":55,"title":445,"description":446,"coverAlt":447},"llm-rate-limits-retries-provider-fallback",[386,387,388,389,9],"Rate limits","Retries","Fallback","Reliability",[391,392,393,394,395,396,397,398,399,400,401],"llm rate limit handling","429 too many requests llm api","exponential backoff with jitter","anthropic 529 overloaded error","openai rate limit retry","llm provider fallback","litellm fallbacks","retry-after header","llm circuit breaker","multi provider llm failover","claude api rate limits",[403,406,409],{"name":404,"url":405},"Token bucket","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FToken_bucket",{"name":407,"url":408},"Exponential backoff","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FExponential_backoff",{"name":410,"url":411},"Circuit breaker design pattern","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FCircuit_breaker_design_pattern",[413,416,419,422,425,428,431,433,436,439],{"title":414,"url":415},"Claude Platform Docs: Rate limits","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fapi\u002Frate-limits",{"title":417,"url":418},"Claude Platform Docs: API errors","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fapi\u002Ferrors",{"title":420,"url":421},"OpenAI API Docs: Rate limits","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Frate-limits",{"title":423,"url":424},"AWS Builders Library: Timeouts, retries and backoff with jitter","https:\u002F\u002Fbuilder.aws.com\u002Fcontent\u002F3EumjoZascWd1oZiEgL8ORlv3qE\u002Ftimeouts-retries-and-backoff-with-jitter",{"title":426,"url":427},"MDN: Retry-After header","https:\u002F\u002Fdeveloper.mozilla.org\u002Fen-US\u002Fdocs\u002FWeb\u002FHTTP\u002FReference\u002FHeaders\u002FRetry-After",{"title":429,"url":430},"Martin Fowler: Circuit Breaker (2014)","https:\u002F\u002Fmartinfowler.com\u002Fbliki\u002FCircuitBreaker.html",{"title":432,"url":38},"LiteLLM Docs: Router, retries and cooldowns",{"title":434,"url":435},"LiteLLM Docs: Proxy fallbacks and reliability","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Fproxy\u002Freliability",{"title":437,"url":438},"OpenRouter Docs: Model fallbacks","https:\u002F\u002Fopenrouter.ai\u002Fdocs\u002Fguides\u002Frouting\u002Fmodel-fallbacks",{"title":440,"url":441},"AI SDK Docs: Settings (maxRetries, timeout)","https:\u002F\u002Fai-sdk.dev\u002Fdocs\u002Fai-sdk-core\u002Fsettings","\u002Fimages\u002Fblog\u002Fllm-rate-limits-retries-provider-fallback\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-rate-limits-retries-provider-fallback\u002Fog.jpg",[55,56,57],"LLM rate limits in production: retries, backoff and provider fallback","How LLM rate limits work, which errors to retry, backoff with jitter, retrying in one layer only, and fallback chains that do not break your output.","Small squares climbing three steps up to a wide rectangle, two more rectangles behind it in a row.",{"slug":449,"published":328,"minutes":450,"category":7,"tags":451,"keywords":457,"about":468,"sources":478,"cover":503,"og":504,"expertise":53,"locales":505,"lang":55,"title":506,"description":507,"coverAlt":508},"prompt-versioning-management",8,[452,453,454,455,456],"Prompt management","Versioning","LLMOps","Evals","Langfuse",[458,459,460,461,462,463,464,465,466,467],"prompt versioning","prompt management","prompt registry","version control for prompts","langfuse prompt management","mlflow prompt registry","prompts in git","openai prompt object deprecation","prompt rollback production","prompt ci evaluation",[469,472,475],{"name":470,"url":471},"Prompt engineering","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FPrompt_engineering",{"name":473,"url":474},"Version control","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FVersion_control",{"name":476,"url":477},"Continuous integration","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FContinuous_integration",[479,482,485,488,491,494,497,500],{"title":480,"url":481},"OpenAI API Docs: Migrate from prompt objects","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fprompting\u002Fmigrate-from-prompt-object",{"title":483,"url":484},"OpenAI API Docs: Prompt engineering (reusable prompts deprecation note)","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fprompt-engineering",{"title":486,"url":487},"Langfuse Docs: Prompt version control","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Fprompt-version-control",{"title":489,"url":490},"Langfuse Docs: Prompt caching in the SDKs","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Fcaching",{"title":492,"url":493},"Langfuse Docs: Link prompts to traces","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Flink-to-traces",{"title":495,"url":496},"MLflow Docs: Prompt Registry","https:\u002F\u002Fmlflow.org\u002Fdocs\u002Flatest\u002Fgenai\u002Fprompt-registry\u002F",{"title":498,"url":499},"Promptfoo Docs: GitHub Action","https:\u002F\u002Fwww.promptfoo.dev\u002Fdocs\u002Fintegrations\u002Fgithub-action\u002F",{"title":501,"url":502},"Claude Platform Docs: Prompt caching","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fprompt-caching","\u002Fimages\u002Fblog\u002Fprompt-versioning-management\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fprompt-versioning-management\u002Fog.jpg",[55,56,57],"Prompt versioning in production: git, registries, labels and rollbacks","How to version prompts in production: what belongs in a version, git versus a prompt registry, labels and rollbacks, eval gates in CI and cache side effects.","Five upright cards in a row, the third turned back on a curved arrow.",{"slug":510,"published":328,"minutes":511,"category":7,"tags":512,"keywords":517,"about":528,"sources":538,"cover":559,"og":560,"expertise":53,"locales":561,"lang":55,"title":562,"description":563,"coverAlt":564},"llm-batch-api-offline-workloads",9,[513,514,454,515,516],"Batch API","Cost","Prompt caching","Data pipelines",[518,519,520,521,522,523,524,525,526,527],"llm batch api","anthropic message batches api","openai batch api","gemini batch mode","batch inference llm cost","batch api 50 percent discount","openai flex processing","bedrock batch inference","llm offline processing","batch api prompt caching",[529,532,535],{"name":530,"url":531},"Batch processing","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FBatch_processing",{"name":533,"url":534},"JSON Lines","https:\u002F\u002Fjsonlines.org\u002F",{"name":536,"url":537},"Large language model","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FLarge_language_model",[539,542,544,545,548,551,553,556],{"title":540,"url":541},"Claude Platform Docs: Batch processing","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fbatch-processing",{"title":543,"url":415},"Claude Platform Docs: Rate limits (Message Batches API)",{"title":501,"url":502},{"title":546,"url":547},"OpenAI API Docs: Batch API","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fbatch",{"title":549,"url":550},"OpenAI API Docs: Flex processing","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fflex-processing",{"title":552,"url":421},"OpenAI API Docs: Rate limits (batch queue limits)",{"title":554,"url":555},"Google AI for Developers: Gemini Batch API","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fbatch-api",{"title":557,"url":558},"Amazon Bedrock User Guide: Batch inference","https:\u002F\u002Fdocs.aws.amazon.com\u002Fbedrock\u002Flatest\u002Fuserguide\u002Fbatch-inference.html","\u002Fimages\u002Fblog\u002Fllm-batch-api-offline-workloads\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-batch-api-offline-workloads\u002Fog.jpg",[55,56,57],"LLM batch APIs: half-price inference for work that can wait","When to use the Anthropic, OpenAI and Gemini batch APIs, how limits, result matching and failures work, and how batch discounts stack with prompt caching.","A wide box holding eight small squares, tiny squares arriving from the left, one long arrow returning.",1792261156625]