[{"data":1,"prerenderedAt":633},["ShallowReactive",2],{"blog-arize-phoenix-en":3},{"slug":4,"published":5,"minutes":6,"category":7,"tags":8,"keywords":14,"about":22,"sources":31,"cover":59,"og":60,"expertise":61,"locales":62,"lang":63,"title":66,"description":67,"coverAlt":68,"metaTitle":69,"takeaways":70,"faq":76,"toc":89,"blocks":114,"others":392},"arize-phoenix","2026-07-03",10,"llmops",[9,10,11,12,13],"LLM observability","Tracing","OpenTelemetry","Self-hosted","Evaluation",[15,16,17,18,19,20,21],"arize phoenix","phoenix arize self-hosted","arize phoenix docker","llm tracing opentelemetry","phoenix vs langfuse","openinference instrumentation","llm observability tools",[23,26,28],{"name":24,"url":25},"Arize Phoenix","https:\u002F\u002Fgithub.com\u002FArize-ai\u002Fphoenix",{"name":11,"url":27},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FOpenTelemetry",{"name":29,"url":30},"Observability (software)","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FObservability_(software)",[32,35,38,41,44,47,50,53,56],{"title":33,"url":34},"Arize Phoenix: open-source AI observability and evaluation","https:\u002F\u002Fphoenix.arize.com",{"title":36,"url":37},"Phoenix documentation: self-hosting","https:\u002F\u002Farize.com\u002Fdocs\u002Fphoenix\u002Fself-hosting",{"title":39,"url":40},"Phoenix documentation: licence, Elastic License 2.0","https:\u002F\u002Farize.com\u002Fdocs\u002Fphoenix\u002Fself-hosting\u002Flicense",{"title":42,"url":43},"Phoenix documentation: architecture, storage and scaling","https:\u002F\u002Farize.com\u002Fdocs\u002Fphoenix\u002Fself-hosting\u002Farchitecture",{"title":45,"url":46},"Phoenix documentation: setup tracing","https:\u002F\u002Farize.com\u002Fdocs\u002Fphoenix\u002Ftracing\u002Fhow-to-tracing\u002Fsetup-tracing",{"title":48,"url":49},"Phoenix documentation: OpenTelemetry SDK setup","https:\u002F\u002Farize.com\u002Fdocs\u002Fphoenix\u002Ftracing\u002Fhow-to-tracing\u002Fsetup-tracing\u002Fsetup-using-phoenix-otel",{"title":51,"url":52},"Arize pricing: AX Free, AX Pro and AX Enterprise","https:\u002F\u002Farize.com\u002Fpricing\u002F",{"title":54,"url":55},"Langfuse pricing: cloud plans and billable units","https:\u002F\u002Flangfuse.com\u002Fpricing",{"title":57,"url":58},"LangChain pricing: LangSmith plans","https:\u002F\u002Fwww.langchain.com\u002Fpricing","\u002Fimages\u002Fblog\u002Farize-phoenix\u002Fcover.webp","\u002Fimages\u002Fblog\u002Farize-phoenix\u002Fog.jpg","ai-engineer",[63,64,65],"en","de","hu","Arize Phoenix review: LLM tracing and evals you host yourself","Arize Phoenix is an ELv2-licensed tracing and evaluation server you run on your own database. What it does well, what it costs in operations, and where Langfuse, Braintrust and LangSmith beat it.","Three bars of decreasing width stacked on a cylinder, each ending in a small circle.","Arize Phoenix: tracing you run yourself · Balázs Csorba",[71,72,73,74,75],"Phoenix is free with no span caps and no feature gates: the Elastic License 2.0 covers the whole self-hosted platform, and the paid meter is Arize AX rather than Phoenix.","The deployment is one container plus a database, SQLite by default and PostgreSQL 14 or newer for production, and the architecture docs state that a single instance is one tenant.","Ingestion is standard OTLP with OpenInference attributes, so the exporter can be swapped without a proprietary SDK in the request path.","Arize AX Free allows 25,000 spans and 1 GB a month with 15 days of retention, and AX Pro costs $50 a month for 50,000 spans, 10 GB and 30 days.","The trade is operational: no support contract, no uptime commitment and no multi-tenancy below AX Enterprise, so retention, backups and access control stay with the adopter.",[77,80,83,86],{"q":78,"a":79},"Is Arize Phoenix really free?","The self-hosted platform is free under the Elastic License 2.0, with no span limits, no usage metering and no feature gates; you pay only for the infrastructure it runs on. The licence forbids offering Phoenix as a hosted or managed service to third parties. Paid tiers exist only in Arize AX: Free at $0, Pro at $50 a month and Enterprise by quotation.",{"q":81,"a":82},"What database does Phoenix use?","SQLite by default, writing into the working directory, which suits a single developer. For production the architecture docs recommend PostgreSQL with a minimum supported version of 14, set through the database URL environment variable. Several instances can share one database behind a load balancer, or teams can be isolated with separate databases or separate PostgreSQL schemas.",{"q":84,"a":85},"How does Phoenix receive traces?","Applications export spans over OTLP using the OpenInference semantic conventions, either through phoenix.otel.register in Python or the TypeScript package for Node. The server answers on port 6006 locally, and auto_instrument=True activates whichever OpenInference instrumentor packages are installed in the environment.",{"q":87,"a":88},"Phoenix or Langfuse?","Phoenix is the pick when traces must stay inside your own network, including air-gapped deployments, because nothing in the open build talks to Arize and there is no usage meter. Langfuse is the pick when a team wants one maintained platform with prompt management and a $29 cloud tier instead of operating the stack itself.",[90,93,96,99,102,105,108,111],{"id":91,"title":92},"what-it-is","What it is",{"id":94,"title":95},"how-it-works","How it works",{"id":97,"title":98},"getting-started","Getting started",{"id":100,"title":101},"self-hosting","Self-hosting in practice",{"id":103,"title":104},"pricing","What it costs",{"id":106,"title":107},"where-it-shingles","Where it falls short",{"id":109,"title":110},"verdict","Verdict",{"id":112,"title":113},"sources","Sources",[115,119,122,125,132,150,153,154,157,163,170,171,182,184,195,209,210,217,232,235,236,239,277,280,281,284,330,333,338,339,342,354,360,361],{"type":116,"content":117},"paragraph",[118],"Arize Phoenix is an open-source server that records what an LLM application did — every prompt, retrieval step, tool call and token — and then scores it. It starts from a single command on a laptop or lands in your own Kubernetes cluster, and the licence puts no meter on any of it. The position of this review is blunt: Phoenix is the shortest path to keeping trace data inside your network, and what you pay for that is the job of running an observability stack yourself.",{"type":116,"content":120},[121],"It sits between the application SDK and the dashboard, competing with Langfuse, Braintrust and LangSmith for that slot, and with Arize's own AX cloud for teams that would rather send an invoice than operate a database. What it usually replaces is worse: a pile of log dumps, a dashboard nobody reads and a spreadsheet of evaluation results.",{"type":123,"level":124,"id":91,"text":92},"heading",2,{"type":116,"content":126},[127,128,131],"Phoenix is a containerised application in three parts — a web interface, a trace collector and a SQL backend — released by Arize under the Elastic License 2.0. The Python package is ",{"tag":129,"children":130},"code",[4],", at version 20.19.0 in early October 2026 and requiring Python 3.11 or newer, and the repository holds a little over 11,700 stars. Everything is in the free build: tracing, annotation, datasets, experiments, a prompt IDE and LLM-as-a-judge evaluation.",{"type":133,"ordered":134,"items":135},"list",false,[136,138,140,142,144,146,148],[137],"Licence: Elastic License 2.0 (ELv2), free to self-host with no usage limits and no feature gates",[139],"Ingestion: OTLP with OpenInference semantic conventions, plus auto-instrumentation packages for providers and frameworks",[141],"Storage: SQLite by default, PostgreSQL 14 or newer for production, both behind the same SQL schema",[143],"Deployment: terminal, Docker, Kubernetes, Helm, CloudFormation and one-click templates for Railway, Render, Cloud Run and Azure",[145],"Scope: traces and sessions, annotations, datasets, experiments, prompt versioning, code scorers and LLM-as-a-judge",[147],"Tenancy: one tenant per instance, with OAuth2, LDAP, local accounts and role-based access control in the same build",[149],"Cloud counterpart: Arize AX, which is where support, an uptime commitment and multi-tenancy live",{"type":116,"content":151},[152],"For a team whose customer or regulator will not accept traces leaving the country, that list is the shortlist on its own. For everyone else it is a trade: the whole product costs nothing, and the jobs a vendor would otherwise do — retention, backups, upgrades, and the answer to who is paged when the collector stops — become yours.",{"type":123,"level":124,"id":94,"text":95},{"type":116,"content":155},[156],"The instrumented process emits spans to an OpenTelemetry exporter, Phoenix accepts them over OTLP on port 6006, stores them in SQL and groups them into projects, sessions and traces. Each span carries OpenInference attributes for tokens, cost, model, retrieval payloads and tool arguments, which is what later lets a scorer read the run rather than only its final string.",{"type":158,"src":159,"alt":160,"wide":161,"caption":162},"image","\u002Fimages\u002Fblog\u002Farize-phoenix\u002Fdiagram-8.webp","How a trace reaches Phoenix and comes back as an experiment An instrumented agent application sends spans through an OpenTelemetry exporter and OpenInference instrumentation to the Phoenix collector on port 6006. The collector writes to SQLite or PostgreSQL and serves the same data through the web interface. A return path takes traces into datasets, experiments and scorers, which feed the next change to the application.",true,"One instance holds collector, storage and interface; the evaluation loop reads the same rows the interface shows.",{"type":116,"content":164},[165,166,169],"The loop after that is the one Arize publishes: observe, annotate, hypothesise, experiment, measure. A production trace becomes a dataset row, a candidate prompt runs against that dataset as an experiment, and scorers — built-in, code-based or judge-based — attach scores to the records the interface already shows. The documentation also describes PXI, an agent interface for investigating issues and running experiments over captured traces, plus an agent-assisted setup command, ",{"tag":129,"children":167},[168],"px setup",", which waits for a real trace before it reports success.",{"type":123,"level":124,"id":97,"text":98},{"type":116,"content":172},[173,174,177,178,181],"Two commands. The server is one Python invocation, ",{"tag":129,"children":175},[176],"uvx arize-phoenix serve",", which answers on http:\u002F\u002Flocalhost:6006 with an empty project. The client side is ",{"tag":129,"children":179},[180],"arize-phoenix-otel"," plus the OpenInference instrumentor for the SDK the application already uses:",{"type":129,"code":183},"# Terminal 1: uvx arize-phoenix serve  ->  UI on http:\u002F\u002Flocalhost:6006\n# pip install \"arize-phoenix-otel>=0.16.0\" openinference-instrumentation-openai\nfrom phoenix.otel import register\n\nregister(\n    project_name=\"support-agent\",\n    auto_instrument=True,\n    endpoint=\"http:\u002F\u002Flocalhost:6006\u002Fv1\u002Ftraces\",\n)\n\nfrom openai import OpenAI\n\nclient = OpenAI()\nreply = client.responses.create(\n    model=\"gpt-5-mini\",\n    input=\"Summarise this ticket in one sentence.\",\n)\nprint(reply.output_text)",{"type":116,"content":185},[186,187,190,191,194],"What comes back is a project with real spans: input, output, model, token counts, latency and the nesting of tool calls under the agent turn that triggered them. The ",{"tag":129,"children":188},[189],"register()"," call reads ",{"tag":129,"children":192},[193],"PHOENIX_COLLECTOR_ENDPOINT"," when it is set, so the same code reaches a laptop, a shared staging server or an air-gapped deployment without an edit.",{"type":196,"variant":197,"title":198,"body":199},"callout","tip","auto_instrument installs nothing",[200],[201,204,205,208],{"tag":129,"children":202},[203],"register(auto_instrument=True)"," only switches on OpenInference instrumentor packages that are already installed in the environment: ",{"tag":129,"children":206},[207],"pip install openinference-instrumentation-openai",", or the langchain, anthropic and llama-index equivalents, for every framework that should appear in the trace. Miss one and that layer is silently absent rather than loudly broken.",{"type":123,"level":124,"id":100,"text":101},{"type":116,"content":211},[212,213,216],"The documentation claims an air-gapped deployment: nothing in the open build talks to Arize, and traces, prompts and datasets stay inside your infrastructure. The footprint behind that claim is one container and one database, and the working directory or ",{"tag":129,"children":214},[215],"PHOENIX_SQL_DATABASE_URL"," is the only stateful part that deserves a backup schedule.",{"type":133,"ordered":134,"items":218},[219,224,226,228,230],[220,221,223],"Storage: SQLite in the working directory by default; setting ",{"tag":129,"children":222},[215]," switches to PostgreSQL, minimum supported version 14",[225],"Images: arizephoenix\u002Fphoenix on Docker Hub with latest, pinned version, nonroot and debug tags, plus a separate arizephoenix\u002Fphoenix-helm chart",[227],"Authentication: OAuth2, LDAP and local accounts, with role-based access control and retention policies per project",[229],"Scale-out: several instances behind a load balancer on one database, or one instance per team with its own database",[231],"Isolation: a schema setting shares one database between teams without sharing rows",{"type":116,"content":233},[234],"Two limits are worth knowing before the first production week. A single instance is one tenant, so team isolation means several deployments, and group-based multi-tenancy sits in the issue tracker as a 2026 item rather than a shipped feature. SQLite is also the development backend: the architecture page routes production traffic to PostgreSQL, which puts backups, migrations and connection pooling in your column.",{"type":123,"level":124,"id":103,"text":104},{"type":116,"content":237},[238],"Phoenix itself has no price. The self-hosting page lists no licence fees, no usage limits and no feature gates, so the monthly cost is the machine, the database and whoever owns them. The commercial surface is Arize AX, a separate managed product whose free tier is enough to evaluate and whose Pro tier is the first real bill:",{"type":240,"head":241,"rows":250},"table",[242,244,246,248],[243],"Plan",[245],"Price",[247],"Spans and storage",[249],"Retention",[251,260,269],[252,254,256,258],[253],"AX Free",[255],"$0",[257],"25,000 spans and 1 GB a month",[259],"15 days",[261,263,265,267],[262],"AX Pro",[264],"$50 a month",[266],"50,000 spans and 10 GB a month",[268],"30 days",[270,272,274,276],[271],"AX Enterprise",[273],"Custom",[275],"Custom span and volume limits",[273],{"type":116,"content":278},[279],"Both of the first two tiers are hosted by Arize, and the pricing table lists self-hosted deployment against AX Enterprise only. Against the field the free self-hosted path is the outlier: Langfuse caps its free cloud at 50,000 units a month, Braintrust at 1 GB and LangSmith at 5,000 traces, while an unlimited Phoenix costs an instance. The catch sits at the far end — dedicated support and an uptime commitment are Enterprise lines, so a team that needs someone else accountable for availability cannot buy that for $50.",{"type":123,"level":124,"id":106,"text":107},{"type":116,"content":282},[283],"Phoenix asks you to be your own observability vendor, and it shows. Retention is whatever you configure, ingestion volume is whatever the disk takes, and nothing in the open build arrives with a support contract or an uptime commitment. The SQL backend is not an analytical engine either: the architecture page routes high-volume, sub-second OLAP work to adb, Arize's proprietary database, which exists only inside AX. The project also ships quickly, which is good for features and awkward for pinning an upgrade window, and the polished hosted extras — managed agents, issue detection, repository access — are AX rows rather than Phoenix rows.",{"type":240,"head":285,"rows":294},[286,288,290,292],[287],"Tool",[289],"Free tier",[291],"Paid entry",[293],"Self-host",[295,303,312,321],[296,297,299,301],[24],[298],"No span caps, local install",[300],"AX Pro $50 a month",[302],"Free under ELv2",[304,306,308,310],[305],"Langfuse",[307],"50,000 units, 30 days, 2 users",[309],"Core $29 a month",[311],"Free, Docker Compose",[313,315,317,319],[314],"Braintrust",[316],"1 GB, 10,000 scores, 14 days",[318],"Pro $249 a month",[320],"Enterprise only",[322,324,326,328],[323],"LangSmith",[325],"1 seat, 5,000 base traces",[327],"Plus $39 a seat",[329],"Enterprise add-on",{"type":116,"content":331},[332],"Choose by constraint rather than by feature count. If data residency is the requirement, Phoenix or self-hosted Langfuse is the answer and Braintrust is not on the list until sales is involved. If the bill matters more than the data plane, Langfuse Core at $29 undercuts AX Pro and carries prompt management with it. If evaluation in CI is the actual job, a scorer library and a ready-made eval action are a shorter route than assembling the parts here.",{"type":334,"content":335},"quote",[336,337],"You may not provide the software to third parties as a hosted or managed service, where the service provides users with access to any substantial set of the features or functionality of the software."," — Elastic License 2.0, limitations",{"type":123,"level":124,"id":109,"text":110},{"type":116,"content":340},[341],"Phoenix is the right default for a team that cannot export traces and has someone who enjoys running Postgres. It is the wrong default for a team that wants an evaluation platform this week with no infrastructure conversation, because storage, retention, access control and upgrades are real work even though the licence is free.",{"type":133,"ordered":161,"items":343},[344,346,348,350,352],[345],"Choose it when trace data must stay in your own network, including air-gapped environments: the open build sends nothing to Arize.",[347],"Choose it when the cost model matters more than the feature checklist: no span caps, no seat fee and no ingestion meter.",[349],"Choose it when OpenTelemetry is already the house standard, since ingestion is OTLP and the exporter can be exchanged without touching the application.",[351],"Do not choose it when you need multi-tenant hosting with an uptime commitment at an entry price: AX Pro is hosted only, and contracts start at Enterprise.",[353],"Do not choose it when evaluation in CI is the primary job, or when nobody on the team wants to own a database.",{"type":196,"variant":355,"title":356,"body":357},"note","Free to run is not free to operate",[358],[359],"Removing the licence bill does not remove the operational one: a database, a retention policy, an upgrade calendar and an owner. Teams that budget for that get the best deal in this category of tooling. Teams that assume it away end up with traces nobody trusts and a dashboard nobody opens.",{"type":123,"level":124,"id":112,"text":113},{"type":133,"ordered":161,"items":362},[363,367,370,374,377,380,383,386,389],[364],{"tag":365,"href":34,"children":366},"a",[33],[368],{"tag":365,"href":37,"children":369},[36],[371],{"tag":365,"href":40,"children":372},[373],"Phoenix documentation: licence under the Elastic License 2.0",[375],{"tag":365,"href":43,"children":376},[42],[378],{"tag":365,"href":46,"children":379},[45],[381],{"tag":365,"href":49,"children":382},[48],[384],{"tag":365,"href":52,"children":385},[51],[387],{"tag":365,"href":55,"children":388},[54],[390],{"tag":365,"href":58,"children":391},[57],[393,450,517,577],{"slug":394,"published":395,"minutes":396,"category":7,"tags":397,"keywords":403,"about":417,"sources":425,"cover":444,"og":445,"expertise":61,"locales":446,"lang":63,"title":447,"description":448,"coverAlt":449},"deepseek-v4-1-flash-1017","2026-10-17",15,[398,399,400,401,402],"DeepSeek","Open-weight models","KV cache","Mixture of experts","LLM cost",[404,405,406,407,408,409,410,411,412,413,414,415,416],"DeepSeek V4.1 Flash 1017","DeepSeek V4.1 Flash","Causal Encoder-Decoder","Compressed Sparse Attention 2","CSA2","FP4 KV cache","KV cache compression","SWA Bounded Replay","Engram conditional memory","DSpark speculative decoding","DeepSeek pricing","DeepSWE benchmark","open-weight coding model",[418,420,422],{"name":398,"url":419},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FDeepSeek",{"name":401,"url":421},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FMixture_of_experts",{"name":423,"url":424},"Transformer (deep learning architecture)","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FTransformer_(deep_learning_architecture)",[426,429,432,435,438,441],{"title":427,"url":428},"DeepSeek-AI: DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression (arXiv:2609.19969)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.19969",{"title":430,"url":431},"Hugging Face: deepseek-ai\u002FDeepSeek-V4.1-Flash-1017 model card","https:\u002F\u002Fhuggingface.co\u002Fdeepseek-ai\u002FDeepSeek-V4.1-Flash-1017",{"title":433,"url":434},"OpenRouter: DeepSeek V4.1 Flash 1017, pricing and providers","https:\u002F\u002Fopenrouter.ai\u002Fdeepseek\u002Fdeepseek-v4.1-flash-1017",{"title":436,"url":437},"The Decoder: Deepseek V4.1 Flash 1017 beats larger rivals in agentic tasks while once again massively cutting per-task costs","https:\u002F\u002Fthe-decoder.com\u002Fdeepseek-v4-1-flash-1017-beats-larger-rivals-in-agentic-tasks-while-once-again-massively-cutting-per-task-costs\u002F",{"title":439,"url":440},"AI Weekly: DeepSeek opens V4.1-Flash-1017 weights and $0.75 Search API","https:\u002F\u002Faiweekly.co\u002Falerts\u002Fdeepseek-opens-v41-flash-1017-weights-and-075-search-api",{"title":442,"url":443},"DeepSeek on X: DeepSeek-V4.1-Flash-1017 announcement","https:\u002F\u002Fx.com\u002Fdeepseek_ai\u002Fstatus\u002F2111375640717033777","\u002Fimages\u002Fblog\u002Fdeepseek-v4-1-flash-1017\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fdeepseek-v4-1-flash-1017\u002Fog.jpg",[63,64,65],"DeepSeek V4.1 Flash 1017: 890 bytes of KV cache per token, and what that buys an agent","A technical deep dive into DeepSeek V4.1 Flash 1017: the encoder-decoder split, CSA2 cache sharing, FP4 KV cache, the benchmarks and the real cost per agent task.","A tall stack of thin cards on the left, an arrow, and the same cards pressed into a stack a quarter of the height on the right.",{"slug":451,"published":395,"minutes":6,"category":7,"tags":452,"keywords":458,"about":470,"sources":480,"cover":511,"og":512,"expertise":61,"locales":513,"lang":63,"title":514,"description":515,"coverAlt":516},"llm-rate-limits-retries-provider-fallback",[453,454,455,456,457],"Rate limits","Retries","Fallback","Reliability","LLM gateway",[459,460,461,462,463,464,465,466,467,468,469],"llm rate limit handling","429 too many requests llm api","exponential backoff with jitter","anthropic 529 overloaded error","openai rate limit retry","llm provider fallback","litellm fallbacks","retry-after header","llm circuit breaker","multi provider llm failover","claude api rate limits",[471,474,477],{"name":472,"url":473},"Token bucket","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FToken_bucket",{"name":475,"url":476},"Exponential backoff","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FExponential_backoff",{"name":478,"url":479},"Circuit breaker design pattern","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FCircuit_breaker_design_pattern",[481,484,487,490,493,496,499,502,505,508],{"title":482,"url":483},"Claude Platform Docs: Rate limits","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fapi\u002Frate-limits",{"title":485,"url":486},"Claude Platform Docs: API errors","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fapi\u002Ferrors",{"title":488,"url":489},"OpenAI API Docs: Rate limits","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Frate-limits",{"title":491,"url":492},"AWS Builders Library: Timeouts, retries and backoff with jitter","https:\u002F\u002Fbuilder.aws.com\u002Fcontent\u002F3EumjoZascWd1oZiEgL8ORlv3qE\u002Ftimeouts-retries-and-backoff-with-jitter",{"title":494,"url":495},"MDN: Retry-After header","https:\u002F\u002Fdeveloper.mozilla.org\u002Fen-US\u002Fdocs\u002FWeb\u002FHTTP\u002FReference\u002FHeaders\u002FRetry-After",{"title":497,"url":498},"Martin Fowler: Circuit Breaker (2014)","https:\u002F\u002Fmartinfowler.com\u002Fbliki\u002FCircuitBreaker.html",{"title":500,"url":501},"LiteLLM Docs: Router, retries and cooldowns","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Frouting",{"title":503,"url":504},"LiteLLM Docs: Proxy fallbacks and reliability","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Fproxy\u002Freliability",{"title":506,"url":507},"OpenRouter Docs: Model fallbacks","https:\u002F\u002Fopenrouter.ai\u002Fdocs\u002Fguides\u002Frouting\u002Fmodel-fallbacks",{"title":509,"url":510},"AI SDK Docs: Settings (maxRetries, timeout)","https:\u002F\u002Fai-sdk.dev\u002Fdocs\u002Fai-sdk-core\u002Fsettings","\u002Fimages\u002Fblog\u002Fllm-rate-limits-retries-provider-fallback\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-rate-limits-retries-provider-fallback\u002Fog.jpg",[63,64,65],"LLM rate limits in production: retries, backoff and provider fallback","How LLM rate limits work, which errors to retry, backoff with jitter, retrying in one layer only, and fallback chains that do not break your output.","Small squares climbing three steps up to a wide rectangle, two more rectangles behind it in a row.",{"slug":518,"published":395,"minutes":519,"category":7,"tags":520,"keywords":525,"about":536,"sources":546,"cover":571,"og":572,"expertise":61,"locales":573,"lang":63,"title":574,"description":575,"coverAlt":576},"prompt-versioning-management",8,[521,522,523,524,305],"Prompt management","Versioning","LLMOps","Evals",[526,527,528,529,530,531,532,533,534,535],"prompt versioning","prompt management","prompt registry","version control for prompts","langfuse prompt management","mlflow prompt registry","prompts in git","openai prompt object deprecation","prompt rollback production","prompt ci evaluation",[537,540,543],{"name":538,"url":539},"Prompt engineering","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FPrompt_engineering",{"name":541,"url":542},"Version control","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FVersion_control",{"name":544,"url":545},"Continuous integration","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FContinuous_integration",[547,550,553,556,559,562,565,568],{"title":548,"url":549},"OpenAI API Docs: Migrate from prompt objects","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fprompting\u002Fmigrate-from-prompt-object",{"title":551,"url":552},"OpenAI API Docs: Prompt engineering (reusable prompts deprecation note)","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fprompt-engineering",{"title":554,"url":555},"Langfuse Docs: Prompt version control","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Fprompt-version-control",{"title":557,"url":558},"Langfuse Docs: Prompt caching in the SDKs","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Fcaching",{"title":560,"url":561},"Langfuse Docs: Link prompts to traces","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Flink-to-traces",{"title":563,"url":564},"MLflow Docs: Prompt Registry","https:\u002F\u002Fmlflow.org\u002Fdocs\u002Flatest\u002Fgenai\u002Fprompt-registry\u002F",{"title":566,"url":567},"Promptfoo Docs: GitHub Action","https:\u002F\u002Fwww.promptfoo.dev\u002Fdocs\u002Fintegrations\u002Fgithub-action\u002F",{"title":569,"url":570},"Claude Platform Docs: Prompt caching","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fprompt-caching","\u002Fimages\u002Fblog\u002Fprompt-versioning-management\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fprompt-versioning-management\u002Fog.jpg",[63,64,65],"Prompt versioning in production: git, registries, labels and rollbacks","How to version prompts in production: what belongs in a version, git versus a prompt registry, labels and rollbacks, eval gates in CI and cache side effects.","Five upright cards in a row, the third turned back on a curved arrow.",{"slug":578,"published":395,"minutes":579,"category":7,"tags":580,"keywords":585,"about":596,"sources":606,"cover":627,"og":628,"expertise":61,"locales":629,"lang":63,"title":630,"description":631,"coverAlt":632},"llm-batch-api-offline-workloads",9,[581,582,523,583,584],"Batch API","Cost","Prompt caching","Data pipelines",[586,587,588,589,590,591,592,593,594,595],"llm batch api","anthropic message batches api","openai batch api","gemini batch mode","batch inference llm cost","batch api 50 percent discount","openai flex processing","bedrock batch inference","llm offline processing","batch api prompt caching",[597,600,603],{"name":598,"url":599},"Batch processing","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FBatch_processing",{"name":601,"url":602},"JSON Lines","https:\u002F\u002Fjsonlines.org\u002F",{"name":604,"url":605},"Large language model","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FLarge_language_model",[607,610,612,613,616,619,621,624],{"title":608,"url":609},"Claude Platform Docs: Batch processing","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fbatch-processing",{"title":611,"url":483},"Claude Platform Docs: Rate limits (Message Batches API)",{"title":569,"url":570},{"title":614,"url":615},"OpenAI API Docs: Batch API","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fbatch",{"title":617,"url":618},"OpenAI API Docs: Flex processing","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fflex-processing",{"title":620,"url":489},"OpenAI API Docs: Rate limits (batch queue limits)",{"title":622,"url":623},"Google AI for Developers: Gemini Batch API","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fbatch-api",{"title":625,"url":626},"Amazon Bedrock User Guide: Batch inference","https:\u002F\u002Fdocs.aws.amazon.com\u002Fbedrock\u002Flatest\u002Fuserguide\u002Fbatch-inference.html","\u002Fimages\u002Fblog\u002Fllm-batch-api-offline-workloads\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-batch-api-offline-workloads\u002Fog.jpg",[63,64,65],"LLM batch APIs: half-price inference for work that can wait","When to use the Anthropic, OpenAI and Gemini batch APIs, how limits, result matching and failures work, and how batch discounts stack with prompt caching.","A wide box holding eight small squares, tiny squares arriving from the left, one long arrow returning.",1792261156345]