[{"data":1,"prerenderedAt":631},["ShallowReactive",2],{"blog-langsmith-en":3},{"slug":4,"published":5,"minutes":6,"category":7,"tags":8,"keywords":14,"about":22,"sources":31,"cover":53,"og":54,"expertise":55,"locales":56,"lang":57,"title":60,"description":61,"coverAlt":62,"metaTitle":63,"takeaways":64,"faq":70,"toc":86,"blocks":111,"others":391},"langsmith","2026-05-25",10,"llmops",[9,10,11,12,13],"Tracing","LLM evals","OpenTelemetry","Datasets","Retries",[4,15,16,17,18,19,20,21],"langsmith vs langfuse","llm observability tools","llm tracing cost","langsmith pricing","opentelemetry llm tracing","llm evaluation datasets","phoenix vs langsmith",[23,26,28],{"name":24,"url":25},"LangSmith","https:\u002F\u002Fwww.langchain.com\u002Flangsmith",{"name":11,"url":27},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FOpenTelemetry",{"name":29,"url":30},"ClickHouse","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FClickHouse",[32,35,38,41,44,47,50],{"title":33,"url":34},"LangSmith pricing","https:\u002F\u002Fwww.langchain.com\u002Fpricing",{"title":36,"url":37},"LangSmith documentation: administration overview","https:\u002F\u002Fdocs.langchain.com\u002Flangsmith\u002Fadministration-overview",{"title":39,"url":40},"LangSmith documentation: custom instrumentation","https:\u002F\u002Fdocs.langchain.com\u002Flangsmith\u002Fannotate-code",{"title":42,"url":43},"LangSmith documentation: trace with OpenTelemetry","https:\u002F\u002Fdocs.langchain.com\u002Flangsmith\u002Ftrace-with-opentelemetry",{"title":45,"url":46},"LangSmith documentation: evaluation concepts","https:\u002F\u002Fdocs.langchain.com\u002Flangsmith\u002Fevaluation-concepts",{"title":48,"url":49},"LangSmith documentation: self-hosted deployment","https:\u002F\u002Fdocs.langchain.com\u002Flangsmith\u002Fself-hosted",{"title":51,"url":52},"LangChain: introducing end-to-end OpenTelemetry support in LangSmith","https:\u002F\u002Fwww.langchain.com\u002Fblog\u002Fend-to-end-opentelemetry-langsmith","\u002Fimages\u002Fblog\u002Flangsmith\u002Fcover.webp","\u002Fimages\u002Fblog\u002Flangsmith\u002Fog.jpg","ai-engineer",[57,58,59],"en","de","hu","LangSmith: an observability product that grew into an agent platform","LangSmith review: per-trace billing, 14 and 180 day retention, OpenTelemetry ingestion, offline and online evals, and where the platform's pull towards an agent runtime shows up.","Four sheets stacked wider towards the top, with a branching shape feeding the base.","LangSmith review: tracing, evals and the bill · Balázs Csorba",[65,66,67,68,69],"The billable unit is the trace, not the span or the token. Developer includes 5,000 base traces a month for one seat, Plus costs $39 a seat with 10,000 included, and the Developer plan without a payment method on file is capped at 5,000 traces a month.","Base traces are retained for 14 days and extended traces for 180, and online evaluators, automation rules and API feedback with extend_trace_retention upgrade traces to the more expensive tier unless you opt out.","Ingest limits are hourly and per plan: 50,000 events and 500 MB an hour on Developer without payment details, 250,000 events and 2.5 GB with them, and 500,000 events and 5 GB on Plus.","The rest of the platform is billed in LangChain Standard Units at $1.00 each, including Engine, Fleet, Sandboxes and the LLM Gateway, which makes a seat price a poor guide to the invoice.","OpenTelemetry ingestion works through the SDK or an OTLP endpoint, but a span whose parent never arrives is buffered and then silently dropped, which is the failure mode to watch in a partial fan-out.",[71,74,77,80,83],{"q":72,"a":73},"How much does LangSmith cost for a small team?","The Developer plan is free for one seat with 5,000 base traces a month included, and without a payment method on file it is also capped at 5,000 traces a month. Plus is $39 per seat a month with 10,000 base traces included and unlimited extra seats. Enterprise is quoted, and it is the only tier that offers workspace role-based access control and self-hosted deployment.",{"q":75,"a":76},"Do I need LangChain to use LangSmith?","No. The langsmith SDK has a traceable decorator for Python, TypeScript, Kotlin and Java, plus a low-level RunTree API and a REST ingest path, so any code can be traced. LangChain and LangGraph just get the instrumentation for free. LangSmith also ingests OpenTelemetry spans, which the docs recommend for applications that already emit OTLP.",{"q":78,"a":79},"What happens to my traces after two weeks?","Base retention is 14 days, after which traces are no longer reachable in the UI or the API and the associated inputs and outputs are deleted within a day, while some trace metadata is kept for analytics and billing. Extended retention is 180 days and costs more; since 14 September 2026 that 180 days is the maximum for SaaS customers.",{"q":81,"a":82},"Can LangSmith run inside my own infrastructure?","Yes, but only as an Enterprise add-on with a licence key. A self-hosted instance runs the frontend, backend, platform backend, playground, queue and a code-execution service on top of ClickHouse for traces, PostgreSQL for operational data and Redis or Valkey for queues, with optional blob storage. LangChain recommends external database services in production rather than the bundled ones.",{"q":84,"a":85},"Will LangSmith train on my data?","No. The pricing FAQ states that LangSmith does not use your data to train models and that traces, prompts and outputs stay private to your organisation. That is a contractual answer rather than a technical control, so a self-hosted deployment is the option for teams who need the data inside their own perimeter.",[87,90,93,96,99,102,105,108],{"id":88,"title":89},"what-it-is","What it is",{"id":91,"title":92},"how-it-works","How it works",{"id":94,"title":95},"getting-started","Getting started",{"id":97,"title":98},"what-it-costs","What it costs",{"id":100,"title":101},"instrumentation-choices","Native tracing or OpenTelemetry",{"id":103,"title":104},"where-it-shingles","Where it shingle",{"id":106,"title":107},"verdict","Verdict",{"id":109,"title":110},"sources","Sources",[112,116,119,122,125,141,142,145,151,154,155,162,165,176,182,193,194,197,251,254,257,266,267,276,278,281,290,291,294,340,343,344,347,359,365,366],{"type":113,"content":114},"paragraph",[115],"LangSmith is a hosted tracing and evaluation platform for LLM applications: every call your agent makes becomes a tree of runs with inputs, outputs, latency and token counts that you can search, score and compare. The position after reading the documentation: it is the most capable tool of its kind, and the most expensive one to understand, because it is no longer only an observability product.",{"type":113,"content":117},[118],"It competes with Langfuse, Arize Phoenix and Helicone for the observability budget, and increasingly with its own runtime, because the same vendor now sells Deployment, Studio, the Engine, Fleet, Sandboxes and an LLM Gateway on the same pricing page. That expansion is the single most important thing to weigh, since each extra service is metered separately.",{"type":120,"level":121,"id":88,"text":89},"heading",2,{"type":113,"content":123},[124],"The platform has four parts that share one bill: tracing, evaluation, prompt management and an agent runtime. The resource model matters more than the features. An organisation holds workspaces, workspaces hold tracing projects, datasets, annotation queues and prompts, and every trace lives in one project. The core facts:",{"type":126,"ordered":127,"items":128},"list",false,[129,131,133,135,137,139],[130],"A trace is one execution, made of nested runs; a run is created and then updated as the work progresses, which is why an event limit and a trace limit are different numbers.",[132],"Tracing works through the langsmith SDK decorators, a REST ingest API or OpenTelemetry spans from any instrumented application.",[134],"Offline evals run against datasets of examples with reference outputs; online evals score live traces without references.",[136],"Evaluators can be code, LLM-as-judge, a typed decision model, pairwise or human, and one evaluator can be attached to several projects.",[138],"Annotation queues, dataset versions and splits, a prompt registry with commit tags and a playground are all included.",[140],"Self-hosted deployment exists, but only as an Enterprise add-on behind a licence key.",{"type":120,"level":121,"id":91,"text":92},{"type":113,"content":143},[144],"Instrumentation runs in your process and posts runs to LangSmith over HTTPS. The SDK sends from a background thread and batches up to 100 runs from one session into a single API call, so tracing does not sit in the request path. A server-side queue then handles ingestion, retries and integrity checks before writing into the trace store.",{"type":146,"src":147,"alt":148,"wide":149,"caption":150},"image","\u002Fimages\u002Fblog\u002Flangsmith\u002Fdiagram-7.webp","The path of a LangSmith trace Traced application code creates runs. The SDK posts them in batches from a background thread, because a rate limit stops the first 5000 posts to the runs endpoint in a minute. An ingest queue retries and stores the runs in ClickHouse. Dashboards, monitors and the query API read from that store.",true,"Because the queue is asynchronous, a trace that is accepted with a 200 can still fail to arrive, and a short-lived process can exit before its runs are posted.",{"type":113,"content":152},[153],"That queue is why the ingestion limits are expressed as a fixed window rather than a smooth average, and why a 429 is a normal event to handle rather than an outage. The load balancer enforces fixed per-minute caps on every plan: 5000 POST or PATCH requests to the runs endpoints, 5000 to feedbacks, 2000 for any other endpoint, and 30 deletes. The SDK batches, which is what keeps a busy application under those numbers.",{"type":120,"level":121,"id":94,"text":95},{"type":113,"content":156},[157,158,159,160,161],"Two environment variables switch tracing on without touching code, which matters for local development: ","LANGSMITH_TRACING"," gates the decorator and the context manager, and ","LANGSMITH_PROJECT"," names the destination project, defaulting to default. The minimal Python setup traces a pipeline as nested runs:",{"type":163,"code":164},"code","import asyncio\n\nfrom langsmith import Client, traceable\nfrom openai import AsyncOpenAI\n\nclient = Client()\nllm = AsyncOpenAI()\n\n\n@traceable(run_type=\"retriever\", name=\"retrieve_docs\")\nasync def retrieve_docs(question: str) -> list[str]:\n    return [\"Annual report: revenue up 12 percent.\"]\n\n\n@traceable(run_type=\"llm\", name=\"answer\")\nasync def answer(question: str, context: list[str]) -> str:\n    reply = await llm.chat.completions.create(\n        model=\"gpt-5.4-mini\",\n        messages=[{\"role\": \"user\", \"content\": f\"{question}\\n{chr(10).join(context)}\"}],\n    )\n    return reply.choices[0].message.content\n\n\n@traceable(name=\"support_agent\")\nasync def support_agent(question: str) -> str:\n    return await answer(question, await retrieve_docs(question))\n\n\nasync def main() -> None:\n    try:\n        print(await support_agent(\"How did revenue move?\"))\n    finally:\n        await client.flush()  # background thread must finish before exit\n\n\nasyncio.run(main())\n",{"type":113,"content":166},[167,168,169,170,171,172,173,174,175],"The decorator propagates context, so the three functions appear as a tree without any manual parent wiring, and ","run_type"," decides how the dashboard renders a node: ","llm"," gives token counts and latency, ","retriever"," and ","tool"," mark the other kinds of step. The same client runs offline evaluations against a dataset, which is where the value starts to compound.",{"type":113,"content":177},[178,179,180,181],"The same API runs the eval loop. ","evaluate"," takes a target function, a dataset and a list of evaluators, produces an experiment with a run per example, and can be driven from CI:"," Evaluate each committed prompt change against the tagged dataset version and fail the build if the groundedness score drops by more than a point. LangSmith versions datasets automatically when examples change, so a tag can pin a CI run to one state of the data. The documentation is blunt about the starting point: write five to ten curated examples of good output before writing any evaluator.",{"type":183,"variant":184,"title":185,"body":186},"callout","warn","The trace tree is not in your process",[187],[188,189,192],"Tracing runs in a background thread so a slow or unavailable LangSmith does not block your request path, which is the right trade. The consequence is that a process can exit before its runs are posted. The SDK has a ",{"tag":163,"children":190},[191],"flush"," method, and the documentation calls it out explicitly for short-lived jobs and serverless handlers. Skipping it is the most common reason a test or a batch job produces an empty trace list.",{"type":120,"level":121,"id":97,"text":98},{"type":113,"content":195},[196],"Seats are the visible price and traces are the metered one. The Developer plan is free for a single seat with 5,000 base traces a month included; Plus is $39 per seat a month with 10,000 included and unlimited extra seats; Enterprise is quoted and adds self-hosted and hybrid deployment, custom SSO, and attribute-based and role-based access control.",{"type":198,"head":199,"rows":210},"table",[200,202,204,206,208],[201],"Plan",[203],"Seats",[205],"Traces included",[207],"Ingest ceiling per hour",[209],"Retention",[211,222,231,242],[212,214,216,218,220],[213],"Developer, no payment details",[215],"1",[217],"5,000 a month, and a monthly cap of 5,000",[219],"50,000 events and 500 MB",[221],"14 days, extended by upgrade",[223,225,226,228,230],[224],"Developer with payment details",[215],[227],"5,000 a month",[229],"250,000 events and 2.5 GB",[221],[232,234,236,238,240],[233],"Plus",[235],"Unlimited, $39 each",[237],"10,000 a month",[239],"500,000 events and 5 GB",[241],"14 or 180 days",[243,245,247,248,249],[244],"Enterprise",[246],"Custom",[246],[246],[250],"Up to 180 days, configurable",{"type":113,"content":252},[253],"An event is the creation or the update of a run, so a run created and then patched inside the same clock hour counts twice against the hourly limit, and a 2 MB run later updated to 3 MB counts 5 MB against the ingest volume limit. That is the mechanic to model in a capacity plan: trace shape, not request count, drives the ceiling.",{"type":113,"content":255},[256],"The rest of the platform is billed in LangChain Standard Units, with one LSU priced at $1.00. The Engine is scheduled every six hours and a single run is estimated at 7 to 45 LSU depending on trace volume and issue count. Fleet includes 7 LSU on Developer and 37 LSU on Plus, Sandboxes 8 LSU, and a perceived-error evaluator run 0.015 LSU. A $39 seat therefore says very little about the invoice once the runtime is in use.",{"type":183,"variant":184,"title":258,"body":259},"Retention upgrades quietly, and there is a catch",[260],[261,262,265],"Base traces live 14 days, extended 180. Online evaluators, automation rules matching any run in a trace, and feedback posted through the API with an explicit ",{"tag":163,"children":263},[264],"extend_trace_retention"," flag all upgrade a trace to the more expensive tier unless you opt out, and since 14 September 2026 the 180-day maximum for SaaS is the ceiling. The nastier edge is the extended-trace usage limit: once it is reached, rule matching, API feedback and annotation queues are switched off entirely, because each of them could create another extended trace.",{"type":120,"level":121,"id":100,"text":101},{"type":113,"content":268},[269,270,271,272,273,274,275],"LangSmith ingests OpenTelemetry spans two ways. With the SDK integration, ","LANGSMITH_OTEL_ENABLED=true"," makes LangChain and LangGraph emit spans through the LangSmith exporter, and ","LANGSMITH_OTEL_ONLY=true"," stops it sending to LangSmith's own format as well. With any other application, point a standard OTLP exporter at the base endpoint ","https:\u002F\u002Fapi.smith.langchain.com\u002Fotel","; regional endpoints exist for EU, APAC and AWS US. The exporter appends the signal path itself, so putting \u002Fv1\u002Ftraces in the base URL gives a 404.",{"type":163,"code":277},"pip install \"langsmith[otel]\"          # needs langsmith >= 0.3.18, 0.4.25 recommended\n\nexport LANGSMITH_TRACING=true\nexport LANGSMITH_OTEL_ENABLED=true\nexport LANGSMITH_ENDPOINT=https:\u002F\u002Fapi.smith.langchain.com\nexport LANGSMITH_API_KEY=...\n\n# fan out one OTLP stream to LangSmith and to the rest of the stack\nexport OTEL_EXPORTER_OTLP_ENDPOINT=https:\u002F\u002Fapi.smith.langchain.com\u002Fotel\nexport OTEL_EXPORTER_OTLP_HEADERS=\"x-api-key=...,Langsmith-Project=support\"\n\n# OTel-only, for teams that do not want a second transport\nexport LANGSMITH_OTEL_ONLY=true\n",{"type":113,"content":279},[280],"The trade is between convenience and overhead. The OTel path costs an attribute-mapping exercise, because span attributes have to be labelled with the langsmith namespace to become run types, run IDs and dotted order; LangChain's own announcement describes the OpenTelemetry route as having slightly higher overhead and recommends the native format when LangSmith is the only destination. The native format also gives pending runs that appear in the UI while the work is still going.",{"type":183,"variant":184,"title":282,"body":283},"A partial fan-out loses spans without an error",[284],[285,286,289],"The OTLP endpoint is asynchronous: it accepts a batch, answers 200 and materialises the runs in the background. A child span whose parent never arrives is buffered and then expires, and because the 200 was already sent, nothing is reported. If part of a distributed trace goes to another backend, the part that reaches LangSmith can quietly lose its parent; on self-hosted installations the buffering window is the Redis setting ",{"tag":163,"children":287},[288],"REDIS_RUNS_EXPIRY_SECONDS",", 12 hours by default.",{"type":120,"level":121,"id":103,"text":104},{"type":113,"content":292},[293],"The weaknesses first, because they decide whether you need this product. The billing unit is the trace, which quietly rewards sampling and punishes an application that traces every request; the load-balancer caps are per service key or personal access token, not per organisation, so a horizontally scaled fleet needs either more keys or the SDK's batching. Workspace role-based access control is Enterprise only, so a growing team on Plus shares one role model. And the resource hierarchy is being rewritten underneath you: workspaces were tenants, agents are in beta as a grouping above projects, and an agent-based workspace addresses traces by agent and environment rather than by project.",{"type":198,"head":295,"rows":304},[296,298,300,302],[297],"Tool",[299],"Primary strength",[301],"Hosting",[303],"What it gives up",[305,313,322,331],[306,307,309,311],[24],[308],"Tracing plus evals plus a managed agent runtime",[310],"Cloud, or self-hosted on Enterprise",[312],"Open core is not an option: the SDK is free, the platform is not",[314,316,318,320],[315],"Langfuse",[317],"MIT core you can self-host and inspect",[319],"Cloud or self-host",[321],"Less of the managed runtime around the traces",[323,325,327,329],[324],"Arize Phoenix",[326],"Apache-2.0, built on OpenTelemetry",[328],"Self-host or cloud",[330],"A narrower product surface around datasets and evals",[332,334,336,338],[333],"Helicone",[335],"Cheap request-level logging with fast setup",[337],"Cloud or proxy deployment",[339],"Less depth on run trees and evaluation workflows",{"type":113,"content":341},[342],"What LangSmith does not give up is measurement: it is the reference implementation for run trees, thread-level conversation views and dataset-driven regression testing, and the annotation queue with reservations is a genuine answer to the question of who labels what. If your application already runs on LangGraph, choosing it is the obvious path and the switching cost is the runtime, not the traces.",{"type":120,"level":121,"id":106,"text":107},{"type":113,"content":345},[346],"LangSmith is worth paying for when an agent's behaviour, not its uptime, is the thing that breaks, and when you need humans in the loop labelling runs and datasets that drive a release gate. It is worth less than its price when you only want a request log, when one trace per user request is too many traces for the budget, or when the platform sprawl makes it unclear what you are paying for.",{"type":126,"ordered":149,"items":348},[349,351,353,355,357],[350],"Adopt it if you need dataset-driven regression tests that run in CI, not just a trace viewer.",[352],"Adopt it if several people have to label runs, because annotation queues with reservations and dataset export are hard to rebuild.",[354],"Adopt it if you already build on LangGraph, and factor the runtime into the decision rather than pretending the traces are separate.",[356],"Be careful if you plan to trace every request: model the event and byte ceilings per hour before you turn instrumentation on in production.",[358],"Look elsewhere if you need self-hosting without a sales conversation, or workspace-level access control, or if a flat price per trace fits your workload better than an LSU-metered platform.",{"type":183,"variant":360,"title":361,"body":362},"note","The short version",[363],[364],"The best trace and evaluation tooling available, sold as a platform that now includes the runtime it observes. Take the tracing and evaluation parts seriously, keep the native transport unless you already run OpenTelemetry, and set usage limits before the first production week rather than after the invoice.",{"type":120,"level":121,"id":109,"text":110},{"type":126,"ordered":149,"items":367},[368,372,375,378,381,384,387],[369],{"tag":370,"href":34,"children":371},"a",[33],[373],{"tag":370,"href":37,"children":374},[36],[376],{"tag":370,"href":40,"children":377},[39],[379],{"tag":370,"href":43,"children":380},[42],[382],{"tag":370,"href":46,"children":383},[45],[385],{"tag":370,"href":49,"children":386},[48],[388],{"tag":370,"href":52,"children":389},[390],"LangChain: end-to-end OpenTelemetry support in LangSmith",[392,449,515,575],{"slug":393,"published":394,"minutes":395,"category":7,"tags":396,"keywords":402,"about":416,"sources":424,"cover":443,"og":444,"expertise":55,"locales":445,"lang":57,"title":446,"description":447,"coverAlt":448},"deepseek-v4-1-flash-1017","2026-10-17",15,[397,398,399,400,401],"DeepSeek","Open-weight models","KV cache","Mixture of experts","LLM cost",[403,404,405,406,407,408,409,410,411,412,413,414,415],"DeepSeek V4.1 Flash 1017","DeepSeek V4.1 Flash","Causal Encoder-Decoder","Compressed Sparse Attention 2","CSA2","FP4 KV cache","KV cache compression","SWA Bounded Replay","Engram conditional memory","DSpark speculative decoding","DeepSeek pricing","DeepSWE benchmark","open-weight coding model",[417,419,421],{"name":397,"url":418},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FDeepSeek",{"name":400,"url":420},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FMixture_of_experts",{"name":422,"url":423},"Transformer (deep learning architecture)","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FTransformer_(deep_learning_architecture)",[425,428,431,434,437,440],{"title":426,"url":427},"DeepSeek-AI: DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression (arXiv:2609.19969)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.19969",{"title":429,"url":430},"Hugging Face: deepseek-ai\u002FDeepSeek-V4.1-Flash-1017 model card","https:\u002F\u002Fhuggingface.co\u002Fdeepseek-ai\u002FDeepSeek-V4.1-Flash-1017",{"title":432,"url":433},"OpenRouter: DeepSeek V4.1 Flash 1017, pricing and providers","https:\u002F\u002Fopenrouter.ai\u002Fdeepseek\u002Fdeepseek-v4.1-flash-1017",{"title":435,"url":436},"The Decoder: Deepseek V4.1 Flash 1017 beats larger rivals in agentic tasks while once again massively cutting per-task costs","https:\u002F\u002Fthe-decoder.com\u002Fdeepseek-v4-1-flash-1017-beats-larger-rivals-in-agentic-tasks-while-once-again-massively-cutting-per-task-costs\u002F",{"title":438,"url":439},"AI Weekly: DeepSeek opens V4.1-Flash-1017 weights and $0.75 Search API","https:\u002F\u002Faiweekly.co\u002Falerts\u002Fdeepseek-opens-v41-flash-1017-weights-and-075-search-api",{"title":441,"url":442},"DeepSeek on X: DeepSeek-V4.1-Flash-1017 announcement","https:\u002F\u002Fx.com\u002Fdeepseek_ai\u002Fstatus\u002F2111375640717033777","\u002Fimages\u002Fblog\u002Fdeepseek-v4-1-flash-1017\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fdeepseek-v4-1-flash-1017\u002Fog.jpg",[57,58,59],"DeepSeek V4.1 Flash 1017: 890 bytes of KV cache per token, and what that buys an agent","A technical deep dive into DeepSeek V4.1 Flash 1017: the encoder-decoder split, CSA2 cache sharing, FP4 KV cache, the benchmarks and the real cost per agent task.","A tall stack of thin cards on the left, an arrow, and the same cards pressed into a stack a quarter of the height on the right.",{"slug":450,"published":394,"minutes":6,"category":7,"tags":451,"keywords":456,"about":468,"sources":478,"cover":509,"og":510,"expertise":55,"locales":511,"lang":57,"title":512,"description":513,"coverAlt":514},"llm-rate-limits-retries-provider-fallback",[452,13,453,454,455],"Rate limits","Fallback","Reliability","LLM gateway",[457,458,459,460,461,462,463,464,465,466,467],"llm rate limit handling","429 too many requests llm api","exponential backoff with jitter","anthropic 529 overloaded error","openai rate limit retry","llm provider fallback","litellm fallbacks","retry-after header","llm circuit breaker","multi provider llm failover","claude api rate limits",[469,472,475],{"name":470,"url":471},"Token bucket","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FToken_bucket",{"name":473,"url":474},"Exponential backoff","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FExponential_backoff",{"name":476,"url":477},"Circuit breaker design pattern","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FCircuit_breaker_design_pattern",[479,482,485,488,491,494,497,500,503,506],{"title":480,"url":481},"Claude Platform Docs: Rate limits","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fapi\u002Frate-limits",{"title":483,"url":484},"Claude Platform Docs: API errors","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fapi\u002Ferrors",{"title":486,"url":487},"OpenAI API Docs: Rate limits","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Frate-limits",{"title":489,"url":490},"AWS Builders Library: Timeouts, retries and backoff with jitter","https:\u002F\u002Fbuilder.aws.com\u002Fcontent\u002F3EumjoZascWd1oZiEgL8ORlv3qE\u002Ftimeouts-retries-and-backoff-with-jitter",{"title":492,"url":493},"MDN: Retry-After header","https:\u002F\u002Fdeveloper.mozilla.org\u002Fen-US\u002Fdocs\u002FWeb\u002FHTTP\u002FReference\u002FHeaders\u002FRetry-After",{"title":495,"url":496},"Martin Fowler: Circuit Breaker (2014)","https:\u002F\u002Fmartinfowler.com\u002Fbliki\u002FCircuitBreaker.html",{"title":498,"url":499},"LiteLLM Docs: Router, retries and cooldowns","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Frouting",{"title":501,"url":502},"LiteLLM Docs: Proxy fallbacks and reliability","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Fproxy\u002Freliability",{"title":504,"url":505},"OpenRouter Docs: Model fallbacks","https:\u002F\u002Fopenrouter.ai\u002Fdocs\u002Fguides\u002Frouting\u002Fmodel-fallbacks",{"title":507,"url":508},"AI SDK Docs: Settings (maxRetries, timeout)","https:\u002F\u002Fai-sdk.dev\u002Fdocs\u002Fai-sdk-core\u002Fsettings","\u002Fimages\u002Fblog\u002Fllm-rate-limits-retries-provider-fallback\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-rate-limits-retries-provider-fallback\u002Fog.jpg",[57,58,59],"LLM rate limits in production: retries, backoff and provider fallback","How LLM rate limits work, which errors to retry, backoff with jitter, retrying in one layer only, and fallback chains that do not break your output.","Small squares climbing three steps up to a wide rectangle, two more rectangles behind it in a row.",{"slug":516,"published":394,"minutes":517,"category":7,"tags":518,"keywords":523,"about":534,"sources":544,"cover":569,"og":570,"expertise":55,"locales":571,"lang":57,"title":572,"description":573,"coverAlt":574},"prompt-versioning-management",8,[519,520,521,522,315],"Prompt management","Versioning","LLMOps","Evals",[524,525,526,527,528,529,530,531,532,533],"prompt versioning","prompt management","prompt registry","version control for prompts","langfuse prompt management","mlflow prompt registry","prompts in git","openai prompt object deprecation","prompt rollback production","prompt ci evaluation",[535,538,541],{"name":536,"url":537},"Prompt engineering","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FPrompt_engineering",{"name":539,"url":540},"Version control","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FVersion_control",{"name":542,"url":543},"Continuous integration","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FContinuous_integration",[545,548,551,554,557,560,563,566],{"title":546,"url":547},"OpenAI API Docs: Migrate from prompt objects","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fprompting\u002Fmigrate-from-prompt-object",{"title":549,"url":550},"OpenAI API Docs: Prompt engineering (reusable prompts deprecation note)","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fprompt-engineering",{"title":552,"url":553},"Langfuse Docs: Prompt version control","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Fprompt-version-control",{"title":555,"url":556},"Langfuse Docs: Prompt caching in the SDKs","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Fcaching",{"title":558,"url":559},"Langfuse Docs: Link prompts to traces","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Flink-to-traces",{"title":561,"url":562},"MLflow Docs: Prompt Registry","https:\u002F\u002Fmlflow.org\u002Fdocs\u002Flatest\u002Fgenai\u002Fprompt-registry\u002F",{"title":564,"url":565},"Promptfoo Docs: GitHub Action","https:\u002F\u002Fwww.promptfoo.dev\u002Fdocs\u002Fintegrations\u002Fgithub-action\u002F",{"title":567,"url":568},"Claude Platform Docs: Prompt caching","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fprompt-caching","\u002Fimages\u002Fblog\u002Fprompt-versioning-management\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fprompt-versioning-management\u002Fog.jpg",[57,58,59],"Prompt versioning in production: git, registries, labels and rollbacks","How to version prompts in production: what belongs in a version, git versus a prompt registry, labels and rollbacks, eval gates in CI and cache side effects.","Five upright cards in a row, the third turned back on a curved arrow.",{"slug":576,"published":394,"minutes":577,"category":7,"tags":578,"keywords":583,"about":594,"sources":604,"cover":625,"og":626,"expertise":55,"locales":627,"lang":57,"title":628,"description":629,"coverAlt":630},"llm-batch-api-offline-workloads",9,[579,580,521,581,582],"Batch API","Cost","Prompt caching","Data pipelines",[584,585,586,587,588,589,590,591,592,593],"llm batch api","anthropic message batches api","openai batch api","gemini batch mode","batch inference llm cost","batch api 50 percent discount","openai flex processing","bedrock batch inference","llm offline processing","batch api prompt caching",[595,598,601],{"name":596,"url":597},"Batch processing","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FBatch_processing",{"name":599,"url":600},"JSON Lines","https:\u002F\u002Fjsonlines.org\u002F",{"name":602,"url":603},"Large language model","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FLarge_language_model",[605,608,610,611,614,617,619,622],{"title":606,"url":607},"Claude Platform Docs: Batch processing","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fbatch-processing",{"title":609,"url":481},"Claude Platform Docs: Rate limits (Message Batches API)",{"title":567,"url":568},{"title":612,"url":613},"OpenAI API Docs: Batch API","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fbatch",{"title":615,"url":616},"OpenAI API Docs: Flex processing","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fflex-processing",{"title":618,"url":487},"OpenAI API Docs: Rate limits (batch queue limits)",{"title":620,"url":621},"Google AI for Developers: Gemini Batch API","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fbatch-api",{"title":623,"url":624},"Amazon Bedrock User Guide: Batch inference","https:\u002F\u002Fdocs.aws.amazon.com\u002Fbedrock\u002Flatest\u002Fuserguide\u002Fbatch-inference.html","\u002Fimages\u002Fblog\u002Fllm-batch-api-offline-workloads\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-batch-api-offline-workloads\u002Fog.jpg",[57,58,59],"LLM batch APIs: half-price inference for work that can wait","When to use the Anthropic, OpenAI and Gemini batch APIs, how limits, result matching and failures work, and how batch discounts stack with prompt caching.","A wide box holding eight small squares, tiny squares arriving from the left, one long arrow returning.",1792261156604]