[{"data":1,"prerenderedAt":815},["ShallowReactive",2],{"blog-helicone-en":3},{"slug":4,"published":5,"minutes":6,"category":7,"tags":8,"keywords":14,"about":22,"sources":28,"cover":56,"og":57,"expertise":58,"locales":59,"lang":60,"title":63,"description":64,"coverAlt":65,"metaTitle":66,"takeaways":67,"faq":73,"toc":86,"blocks":117,"others":574},"helicone","2026-07-03",10,"llmops",[9,10,11,12,13],"LLM observability","OpenTelemetry","Cost tracking","Proxy","Gateway",[15,16,17,18,19,20,21],"helicone llm observability","helicone vs langfuse","llm cost tracking tool","open source llm observability","llm gateway proxy","self-host llm observability","helicone cache",[23,25],{"name":10,"url":24},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FOpenTelemetry",{"name":26,"url":27},"Apache License 2.0","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FApache_License",[29,32,35,38,41,44,47,50,53],{"title":30,"url":31},"Helicone quickstart","https:\u002F\u002Fdocs.helicone.ai\u002Fgetting-started\u002Fquick-start",{"title":33,"url":34},"Helicone AI Gateway overview","https:\u002F\u002Fdocs.helicone.ai\u002Fgateway\u002Foverview",{"title":36,"url":37},"Helicone: latency impact and benchmark","https:\u002F\u002Fdocs.helicone.ai\u002Freferences\u002Flatency-affect",{"title":39,"url":40},"Helicone: proxy versus async integration","https:\u002F\u002Fdocs.helicone.ai\u002Freferences\u002Fproxy-vs-async",{"title":42,"url":43},"Helicone: LLM caching","https:\u002F\u002Fdocs.helicone.ai\u002Ffeatures\u002Fadvanced-usage\u002Fcaching",{"title":45,"url":46},"Helicone: self-hosting with Docker","https:\u002F\u002Fdocs.helicone.ai\u002Fgetting-started\u002Fself-deploy-docker",{"title":48,"url":49},"Helicone: how we calculate cost","https:\u002F\u002Fdocs.helicone.ai\u002Ffaq\u002Fhow-we-calculate-cost",{"title":51,"url":52},"Helicone pricing","https:\u002F\u002Fwww.helicone.ai\u002Fpricing",{"title":54,"url":55},"Helicone repository on GitHub","https:\u002F\u002Fgithub.com\u002FHelicone\u002Fhelicone","\u002Fimages\u002Fblog\u002Fhelicone\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fhelicone\u002Fog.jpg","ai-engineer",[60,61,62],"en","de","hu","Helicone: observability that sits in the request path","Helicone is an Apache-2.0 LLM gateway and observability platform. What the proxy architecture buys, what it costs you, and how the self-hosted stack really looks.","A narrow vertical slab with a rail through it, squares waiting inside and passing out right.","Helicone: LLM observability with a proxy in the path",[68,69,70,71,72],"Helicone is Apache-2.0 with about 6,200 GitHub stars and covers 100 plus models behind one OpenAI-compatible endpoint.","Integration is one line: a different base URL, or the OpenLLMetry async path if the proxy must stay off the critical path.","The proxy adds caching, rate limits, fallbacks and retries, and the async path gives all of those up.","The vendor benchmark reports a mean of 2.21 seconds both direct and proxied on 500 interleaved requests, measured on text-ada-001.","Self-hosting runs five components — web, worker, Jawn, Supabase and ClickHouse plus MinIO — and Jawn no longer proxies, so the gateway is a separate deployment.",[74,77,80,83],{"q":75,"a":76},"Is Helicone open source?","Yes, under Apache 2.0. The repository at roughly 6,200 stars contains the gateway, the log collector and the dashboard. The hosted tiers add features such as SOC 2 reports, HIPAA options, SAML SSO and on-prem deployment terms.",{"q":78,"a":79},"Does the Helicone proxy add latency?","Helicone's own benchmark sends 500 interleaved requests to OpenAI directly and through Helicone and reports a mean of 2.21 seconds in both cases, with p90 at 3.27 versus 3.29 seconds. It is a vendor test on text-ada-001, so treat it as evidence that the overhead is small rather than as a number to plan against.",{"q":81,"a":82},"Can Helicone stay out of the critical path?","Yes, through the OpenLLMetry async integration, which logs after the response and never sits between the application and the provider. The documentation is explicit about the trade: the async path loses bucket caching, custom rate limits and retries.",{"q":84,"a":85},"How does Helicone work with several providers?","The AI Gateway exposes 100 plus models through one OpenAI-compatible endpoint and translates the request to each provider's format. With credits, Helicone holds the provider keys and claims zero markup; you can also bring your own keys.",[87,90,93,96,99,102,105,108,111,114],{"id":88,"title":89},"what-it-is","What it is",{"id":91,"title":92},"how-it-works","How it works",{"id":94,"title":95},"getting-started","Getting started",{"id":97,"title":98},"proxy-or-async","Proxy or async, pick one deliberately",{"id":100,"title":101},"performance","Latency and the cost of a hop",{"id":103,"title":104},"pricing","Pricing and what the meter is",{"id":106,"title":107},"self-hosting","Self-hosting the whole stack",{"id":109,"title":110},"where-it-shingles","Where it falls short",{"id":112,"title":113},"verdict","Verdict",{"id":115,"title":116},"sources","Sources",[118,122,125,128,131,147,148,151,157,160,161,164,167,170,177,178,181,250,253,254,257,326,329,335,336,339,342,410,411,414,428,433,434,437,516,519,520,523,537,543,544],{"type":119,"content":120},"paragraph",[121],"Helicone is an Apache-2.0 licensed LLM gateway and observability platform, and the review position is that the architecture is the whole story: Helicone works by sitting between the application and the model provider, which buys an enormous amount of functionality for one changed line of code and costs a network hop, a data-residency question and a vendor on the critical path. For teams that want a dashboard on Monday, that trade is good. For teams with a latency budget measured in milliseconds or a rule that prompts cannot leave the network, it is the wrong shape.",{"type":119,"content":123},[124],"It competes in two directions at once. Against pure tracing backends such as Langfuse, Arize Phoenix and LangSmith, it wins on time-to-first-dashboard and on the gateway features bolted to the proxy, and loses on OpenTelemetry-native instrumentation and on evaluation tooling. Against API routers such as LiteLLM or OpenRouter, it is observability first with routing attached.",{"type":126,"level":127,"id":88,"text":89},"heading",2,{"type":119,"content":129},[130],"Two products share one codebase. The gateway is an OpenAI-compatible endpoint in front of 100 plus providers, reached by pointing the base URL at ai-gateway.helicone.ai. The observability platform is what records the requests that pass through, stores them in ClickHouse, and answers questions about cost, latency, sessions and users through a dashboard, a SQL-like query language called HQL, alerts, reports and webhooks.",{"type":132,"ordered":133,"items":134},"list",false,[135,137,139,141,143,145],[136],"Licence Apache 2.0, about 6,200 GitHub stars, and a note that Helicone joined Mintlify in 2025.",[138],"One-line integration: change the base URL, or send an async log through the OpenLLMetry instrumentation.",[140],"Cost tracking computed from token usage and a public price database covering more than 300 models and providers.",[142],"Gateway features on the request path: edge cache, custom rate limits by request count, cost or property, and automatic fallbacks.",[144],"Operational surface: sessions, per-user metrics, custom properties, scores, datasets, webhooks and an MCP server over the data.",[146],"Self-hosting that runs five services: a web frontend, a Cloudflare Worker, the Jawn collector, Supabase and ClickHouse, plus MinIO for bodies.",{"type":126,"level":127,"id":91,"text":92},{"type":119,"content":149},[150],"The request path is deliberately thin. Unless a header enables a feature, the worker forwards the request and returns the response untouched; after the response is complete, the proxy ships logs to Kafka for a separate service to consume. The availability documentation states the design intent directly: all business logic falls back to plain proxying on any error, so a bug in observability degrades to a working relay.",{"type":152,"src":153,"alt":154,"wide":155,"caption":156},"image","\u002Fimages\u002Fblog\u002Fhelicone\u002Fdiagram-7.webp","The Helicone request path Top row: the application sends a chat completion through the OpenAI SDK to the Helicone gateway, which sits on Cloudflare Workers and either serves a cache hit or forwards to the provider and returns the response. Bottom row: after the response is complete, logs go to Kafka and then into ClickHouse for analytics and MinIO for request and response bodies, where the dashboard, HQL, alerts and reports read from. Below them a bar: the async path bypasses the gateway entirely, logging after the response from the application itself, and gives up caching, rate limits and retries in exchange.",true,"The gateway is on the critical path only in the proxy mode, and the logging happens after the response in both.",{"type":119,"content":158},[159],"One consequence is worth stating plainly: in proxy mode every prompt and every completion transits a third-party edge network by default. The self-hosted deployment removes that, but it also removes the part of the product that makes Helicone easy, because a self-hosted Jawn no longer proxies at all — the docs record that the gateway routes were removed and the AI Gateway must be deployed separately.",{"type":126,"level":127,"id":94,"text":95},{"type":119,"content":162},[163],"The whole integration is a base URL. The example below also switches on the edge cache, which is the fastest way to see the platform do something visible: the second identical request is served from Cloudflare's KV store rather than from the provider.",{"type":165,"code":166},"code","import os\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url=\"https:\u002F\u002Fai-gateway.helicone.ai\",\n    api_key=os.environ[\"HELICONE_API_KEY\"],\n    default_headers={\n        \"Helicone-Cache-Enabled\": \"true\",\n        \"Cache-Control\": \"max-age=3600\",\n        \"Helicone-Cache-Seed\": \"user-123\",\n        \"Helicone-Property-Env\": \"production\",\n    },\n)\n\nresponse = client.chat.completions.create(\n    model=\"gpt-4o-mini\",\n    messages=[{\"role\": \"user\", \"content\": \"Summarise the refund policy.\"}],\n)\n\nprint(response.choices[0].message.content)\n# The next identical call returns Helicone-Cache: HIT and skips the provider.",{"type":119,"content":168},[169],"The cache key is a hash of the cache seed, the request URL, the whole request body and the relevant headers, which makes it exact-match rather than semantic: change one word or one temperature and it misses. Durations run from an hour to a 365-day maximum, buckets up to 20 stored responses are available for non-deterministic prompts, and Helicone-Cache-Ignore-Keys excludes JSON fields such as a request id or a provider prompt_cache_key from the key.",{"type":171,"variant":172,"title":173,"body":174},"callout","warn","Cache keys leak context",[175],[176],"Because the key includes the request body, a per-user cache seed still derives keys from that user's prompt content, and the responses live in Cloudflare Workers KV rather than in infrastructure you control. For anything containing personal data, leave the cache off and decide deliberately whether Helicone belongs in the request path at all.",{"type":126,"level":127,"id":97,"text":98},{"type":119,"content":179},[180],"This is the decision that shapes everything else. Helicone documents both modes and is unusually honest about the gap between them, which makes the choice easy to reason about even though the answer is rarely comfortable.",{"type":182,"head":183,"rows":190},"table",[184,186,188],[185],"Capability",[187],"Proxy mode",[189],"Async mode",[191,198,204,209,214,219,226,231,236,243],[192,194,196],[193],"On the critical path",[195],"yes",[197],"no",[199,201,202],[200],"Gateway cache and buckets",[195],[203],"not available",[205,207,208],[206],"Custom rate limits",[195],[203],[210,212,213],[211],"Automatic retries",[195],[203],[215,217,218],[216],"Prompt auto-formatting",[195],[203],[220,222,224],[221],"Setup effort",[223],"one base URL",[225],"instrument with OpenLLMetry",[227,229,230],[228],"Sessions and user metrics",[195],[195],[232,234,235],[233],"Custom properties and scores",[195],[195],[237,239,241],[238],"Data path",[240],"prompts cross Helicone's edge",[242],"prompts stay with the provider",[244,246,248],[245],"Use it when",[247],"you want caching and rate limits",[249],"propagation delay is unacceptable",{"type":119,"content":251},[252],"The reasonable conclusion is that these are two products sharing a name. The proxy is a gateway with analytics attached; the async integration is a logging library with a web UI. Teams that take the proxy get caching, rate limiting and fallbacks that they would otherwise build, and pay for it with an extra hop and a data-residency decision. Teams that take the async path keep their latency budget and their network boundary, and build the gateway themselves or go without it.",{"type":126,"level":127,"id":100,"text":101},{"type":119,"content":255},[256],"The published benchmark is small but unusually well specified: 500 requests with unique prompts interleaved between OpenAI and Helicone inside the same one-second window, alternating which endpoint was called first, with the prompt context maximised, on text-ada-001, logging round-trip latency for both sets.",{"type":182,"head":258,"rows":265},[259,261,263],[260],"Statistic",[262],"OpenAI direct (s)",[264],"Helicone proxied (s)",[266,272,279,285,292,299,306,313,320],[267,269,271],[268],"Mean",[270],"2.21",[270],[273,275,277],[274],"Median",[276],"2.87",[278],"2.90",[280,282,284],[281],"Standard deviation",[283],"1.12",[283],[286,288,290],[287],"p90",[289],"3.27",[291],"3.29",[293,295,297],[294],"Maximum",[296],"3.56",[298],"3.76",[300,302,304],[301],"Method",[303],"500 interleaved requests",[305],"same prompts, alternating order",[307,309,311],[308],"Model",[310],"text-ada-001",[312],"text-ada-001, max context",[314,316,318],[315],"Overhead",[317],"baseline",[319],"visible only in the maximum",[321,323,325],[322],"Who ran it",[324],"vendor",[324],{"type":119,"content":327},[328],"Read the maximum row carefully. A 200 millisecond difference on a three-and-a-half second response is 6 per cent, and the same absolute hop applied to a 300 millisecond model call or to a 40 millisecond tool call would dominate it. The benchmark is also a vendor test on a retired model, so it demonstrates that the overhead is bounded, not what it will be for a given workload.",{"type":171,"variant":330,"title":331,"body":332},"note","Where the time actually goes",[333],[334],"Logging is not the cost. The proxy writes to Kafka only after the response is returned, so telemetry does not sit in front of the user. The cost is the round trip to an edge location that is close to the caller but not necessarily close to the provider's region, plus the gateway features you switch on, which by definition do work per request.",{"type":126,"level":127,"id":103,"text":104},{"type":119,"content":337},[338],"The free Hobby plan covers 10,000 requests a month, one seat, one organisation, one gigabyte and seven days of retention. Pro is $79 a month for unlimited seats, one organisation, alerts, reports, HQL, one month of retention and 1,000 ingested logs per minute. Team is $799 for five organisations, SOC 2 and HIPAA options, three months of retention and 15,000 logs per minute. Enterprise adds unlimited organisations, SAML SSO and on-prem deployment.",{"type":119,"content":340},[341],"The number to watch is the one nobody prices prominently: ingestion. A Hobby workspace ingests 10 logs a minute, Pro 1,000, Team 15,000. A production application can emit several logs per user request — a model call, a retrieval step, a tool call — so a moderately busy product exhausts the Pro ceiling before it exhausts the request allowance, and the plan that fixes it costs 799 dollars a month. Storage is metered separately above the first gigabyte.",{"type":182,"head":343,"rows":372},[344,350,356,362,367],[345,346,347,348,349],"Plan","Price","Requests","Ingestion","Retention",[351,352,353,354,355],"Hobby","$0","10,000 per month","10 logs per minute","7 days",[357,358,359,360,361],"Pro","$79 per month","usage-based","1,000 logs per minute","1 month",[363,364,359,365,366],"Team","$799 per month","15,000 logs per minute","3 months",[368,369,359,370,371],"Enterprise","custom","30,000 logs per minute","forever",[373,382,390,401],[374,376,378,380,381],[375],"Seats",[377],"1",[379],"unlimited",[379],[379],[383,385,386,387,389],[384],"Organisations",[377],[377],[388],"5",[379],[391,393,395,397,399],[392],"Notable additions",[394],"nothing",[396],"HQL, alerts, reports",[398],"SOC 2, HIPAA",[400],"SAML SSO, on-prem",[402,404,406,407,408],[403],"Self-host",[405],"free",[405],[405],[409],"Helm chart on request",{"type":126,"level":127,"id":106,"text":107},{"type":119,"content":412},[413],"The Apache-2.0 repository contains the entire platform, and the README is direct about the operational shape: five services, and a manual deployment that it marks as not recommended in favour of the Docker compose file or an Enterprise Helm chart.",{"type":132,"ordered":133,"items":415},[416,418,420,422,424,426],[417],"Web: the dashboard frontend, a Next.js application.",[419],"Worker: the proxy, deployed as Cloudflare Workers in the hosted version.",[421],"Jawn: the log collector, an Express service with Tsoa-generated routes.",[423],"Supabase: the application database and authentication.",[425],"ClickHouse: the analytics store that answers cost and latency questions.",[427],"MinIO: object storage for request and response bodies.",{"type":171,"variant":172,"title":429,"body":430},"Two things the docs get right to warn about",[431],[432],"Jawn no longer proxies: the gateway routes were removed, so a self-hosted deployment runs the AI Gateway as a separate service or does not proxy at all. And port 8585 accepts proxy requests without authentication, so anyone who can reach it can spend your provider credits — restrict it at the firewall, put TLS in front, and mount volumes for Postgres, ClickHouse and MinIO or a restart wipes the data.",{"type":126,"level":127,"id":109,"text":110},{"type":119,"content":435},[436],"The honest weaknesses are structural, not cosmetic. Helicone's data model is built around requests that pass through its gateway, so a team that already traces with OpenTelemetry spans has to choose between two instrumentation stacks. The evaluation story is thin next to Langfuse or Braintrust: prompts, playground and scores exist, but there is no first-class experiments workflow. And the ownership question is live — Helicone joined Mintlify in 2025, which removes the open-core lock-in risk but adds the usual vendor dependency.",{"type":182,"head":438,"rows":447},[439,441,443,445],[440],"",[442],"Helicone",[444],"Langfuse",[446],"Arize Phoenix",[448,457,466,475,484,491,499,507],[449,451,453,455],[450],"Licence",[452],"Apache 2.0",[454],"MIT core, enterprise extras",[456],"Elastic License 2.0",[458,460,462,464],[459],"Instrumentation",[461],"proxy or OpenLLMetry",[463],"OpenTelemetry native",[465],"OpenInference on OTel",[467,469,471,473],[468],"Strongest suit",[470],"one-line start, gateway features",[472],"evaluation and prompt workflows",[474],"local notebook-style evaluation",[476,478,480,482],[477],"Weakest suit",[479],"not OTel-native for tracing",[481],"heavier to start than a base URL",[483],"ELv2 is not permissive",[485,487,489,490],[486],"Gateway features",[488],"cache, limits, fallbacks",[197],[197],[492,493,495,497],[403],[494],"Apache 2.0, five services",[496],"MIT, Docker Compose or Helm",[498],"ELv2, self-hostable",[500,501,503,505],[11],[502],"built in",[504],"configurable per model",[506],"separate product",[508,510,512,514],[509],"Best fit",[511],"teams wanting visibility this week",[513],"teams running evaluations",[515],"teams tracing and evaluating locally",{"type":119,"content":517},[518],"Langfuse is MIT-licensed, was acquired by ClickHouse in January 2026 with the licence and self-hosting explicitly unchanged, and stores traces in ClickHouse as well; if the deciding factor is a permissive licence with a real evaluation stack, it is the stronger tool. Arize Phoenix is built on OpenInference conventions and runs anywhere, including air-gapped, but its Elastic License 2.0 is not a permissive one. Helicone's argument is not capability, it is time: the distance between a repository and a cost dashboard measured in requests per user is one line of code.",{"type":126,"level":127,"id":112,"text":113},{"type":119,"content":521},[522],"Helicone is the right tool when nobody can answer what the LLM spend was last Tuesday, and the wrong tool when latency, data residency or tracing standards decide the architecture. Its engineering is unremarkable in the best sense: the proxy is thin, logs go out after the response, and the product is Apache-2.0 from the gateway to the dashboard. Its weakness is equally plain: it wants to be your gateway, and everything that makes it comfortable is a consequence of that.",{"type":132,"ordered":155,"items":524},[525,527,529,531,533,535],[526],"Use it when the team needs per-request cost, latency and error visibility within a week and has no tracing stack yet.",[528],"Use it when the edge cache, custom rate limits and automatic fallbacks are worth having on the request path.",[530],"Use the async OpenLLMetry path when you want Helicone's analytics without a proxy hop, and accept losing the gateway features.",[532],"Avoid it when prompts must not leave your network or when the proxy hop eats a hard latency budget.",[534],"Avoid it when the platform already traces with OpenTelemetry and a second instrumentation model would fragment the data.",[536],"Reconsider it above roughly 1,000 ingested logs a minute, or when the request allowance is large enough that the usage meter needs its own line in the budget.",{"type":171,"variant":538,"title":539,"body":540},"tip","The one thing to measure",[541],[542],"Before committing, measure the gateway hop against your own latency budget with your own model and region. Helicone publishes a fair benchmark, but a fair benchmark on text-ada-001 tells you almost nothing about a 300 millisecond streaming response served from a region three time zones away from the provider.",{"type":126,"level":127,"id":115,"text":116},{"type":132,"ordered":155,"items":545},[546,550,553,556,559,562,565,568,571],[547],{"tag":548,"href":31,"children":549},"a",[30],[551],{"tag":548,"href":34,"children":552},[33],[554],{"tag":548,"href":37,"children":555},[36],[557],{"tag":548,"href":40,"children":558},[39],[560],{"tag":548,"href":43,"children":561},[42],[563],{"tag":548,"href":46,"children":564},[45],[566],{"tag":548,"href":49,"children":567},[48],[569],{"tag":548,"href":52,"children":570},[51],[572],{"tag":548,"href":55,"children":573},[54],[575,632,699,759],{"slug":576,"published":577,"minutes":578,"category":7,"tags":579,"keywords":585,"about":599,"sources":607,"cover":626,"og":627,"expertise":58,"locales":628,"lang":60,"title":629,"description":630,"coverAlt":631},"deepseek-v4-1-flash-1017","2026-10-17",15,[580,581,582,583,584],"DeepSeek","Open-weight models","KV cache","Mixture of experts","LLM cost",[586,587,588,589,590,591,592,593,594,595,596,597,598],"DeepSeek V4.1 Flash 1017","DeepSeek V4.1 Flash","Causal Encoder-Decoder","Compressed Sparse Attention 2","CSA2","FP4 KV cache","KV cache compression","SWA Bounded Replay","Engram conditional memory","DSpark speculative decoding","DeepSeek pricing","DeepSWE benchmark","open-weight coding model",[600,602,604],{"name":580,"url":601},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FDeepSeek",{"name":583,"url":603},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FMixture_of_experts",{"name":605,"url":606},"Transformer (deep learning architecture)","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FTransformer_(deep_learning_architecture)",[608,611,614,617,620,623],{"title":609,"url":610},"DeepSeek-AI: DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression (arXiv:2609.19969)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.19969",{"title":612,"url":613},"Hugging Face: deepseek-ai\u002FDeepSeek-V4.1-Flash-1017 model card","https:\u002F\u002Fhuggingface.co\u002Fdeepseek-ai\u002FDeepSeek-V4.1-Flash-1017",{"title":615,"url":616},"OpenRouter: DeepSeek V4.1 Flash 1017, pricing and providers","https:\u002F\u002Fopenrouter.ai\u002Fdeepseek\u002Fdeepseek-v4.1-flash-1017",{"title":618,"url":619},"The Decoder: Deepseek V4.1 Flash 1017 beats larger rivals in agentic tasks while once again massively cutting per-task costs","https:\u002F\u002Fthe-decoder.com\u002Fdeepseek-v4-1-flash-1017-beats-larger-rivals-in-agentic-tasks-while-once-again-massively-cutting-per-task-costs\u002F",{"title":621,"url":622},"AI Weekly: DeepSeek opens V4.1-Flash-1017 weights and $0.75 Search API","https:\u002F\u002Faiweekly.co\u002Falerts\u002Fdeepseek-opens-v41-flash-1017-weights-and-075-search-api",{"title":624,"url":625},"DeepSeek on X: DeepSeek-V4.1-Flash-1017 announcement","https:\u002F\u002Fx.com\u002Fdeepseek_ai\u002Fstatus\u002F2111375640717033777","\u002Fimages\u002Fblog\u002Fdeepseek-v4-1-flash-1017\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fdeepseek-v4-1-flash-1017\u002Fog.jpg",[60,61,62],"DeepSeek V4.1 Flash 1017: 890 bytes of KV cache per token, and what that buys an agent","A technical deep dive into DeepSeek V4.1 Flash 1017: the encoder-decoder split, CSA2 cache sharing, FP4 KV cache, the benchmarks and the real cost per agent task.","A tall stack of thin cards on the left, an arrow, and the same cards pressed into a stack a quarter of the height on the right.",{"slug":633,"published":577,"minutes":6,"category":7,"tags":634,"keywords":640,"about":652,"sources":662,"cover":693,"og":694,"expertise":58,"locales":695,"lang":60,"title":696,"description":697,"coverAlt":698},"llm-rate-limits-retries-provider-fallback",[635,636,637,638,639],"Rate limits","Retries","Fallback","Reliability","LLM gateway",[641,642,643,644,645,646,647,648,649,650,651],"llm rate limit handling","429 too many requests llm api","exponential backoff with jitter","anthropic 529 overloaded error","openai rate limit retry","llm provider fallback","litellm fallbacks","retry-after header","llm circuit breaker","multi provider llm failover","claude api rate limits",[653,656,659],{"name":654,"url":655},"Token bucket","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FToken_bucket",{"name":657,"url":658},"Exponential backoff","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FExponential_backoff",{"name":660,"url":661},"Circuit breaker design pattern","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FCircuit_breaker_design_pattern",[663,666,669,672,675,678,681,684,687,690],{"title":664,"url":665},"Claude Platform Docs: Rate limits","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fapi\u002Frate-limits",{"title":667,"url":668},"Claude Platform Docs: API errors","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fapi\u002Ferrors",{"title":670,"url":671},"OpenAI API Docs: Rate limits","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Frate-limits",{"title":673,"url":674},"AWS Builders Library: Timeouts, retries and backoff with jitter","https:\u002F\u002Fbuilder.aws.com\u002Fcontent\u002F3EumjoZascWd1oZiEgL8ORlv3qE\u002Ftimeouts-retries-and-backoff-with-jitter",{"title":676,"url":677},"MDN: Retry-After header","https:\u002F\u002Fdeveloper.mozilla.org\u002Fen-US\u002Fdocs\u002FWeb\u002FHTTP\u002FReference\u002FHeaders\u002FRetry-After",{"title":679,"url":680},"Martin Fowler: Circuit Breaker (2014)","https:\u002F\u002Fmartinfowler.com\u002Fbliki\u002FCircuitBreaker.html",{"title":682,"url":683},"LiteLLM Docs: Router, retries and cooldowns","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Frouting",{"title":685,"url":686},"LiteLLM Docs: Proxy fallbacks and reliability","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Fproxy\u002Freliability",{"title":688,"url":689},"OpenRouter Docs: Model fallbacks","https:\u002F\u002Fopenrouter.ai\u002Fdocs\u002Fguides\u002Frouting\u002Fmodel-fallbacks",{"title":691,"url":692},"AI SDK Docs: Settings (maxRetries, timeout)","https:\u002F\u002Fai-sdk.dev\u002Fdocs\u002Fai-sdk-core\u002Fsettings","\u002Fimages\u002Fblog\u002Fllm-rate-limits-retries-provider-fallback\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-rate-limits-retries-provider-fallback\u002Fog.jpg",[60,61,62],"LLM rate limits in production: retries, backoff and provider fallback","How LLM rate limits work, which errors to retry, backoff with jitter, retrying in one layer only, and fallback chains that do not break your output.","Small squares climbing three steps up to a wide rectangle, two more rectangles behind it in a row.",{"slug":700,"published":577,"minutes":701,"category":7,"tags":702,"keywords":707,"about":718,"sources":728,"cover":753,"og":754,"expertise":58,"locales":755,"lang":60,"title":756,"description":757,"coverAlt":758},"prompt-versioning-management",8,[703,704,705,706,444],"Prompt management","Versioning","LLMOps","Evals",[708,709,710,711,712,713,714,715,716,717],"prompt versioning","prompt management","prompt registry","version control for prompts","langfuse prompt management","mlflow prompt registry","prompts in git","openai prompt object deprecation","prompt rollback production","prompt ci evaluation",[719,722,725],{"name":720,"url":721},"Prompt engineering","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FPrompt_engineering",{"name":723,"url":724},"Version control","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FVersion_control",{"name":726,"url":727},"Continuous integration","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FContinuous_integration",[729,732,735,738,741,744,747,750],{"title":730,"url":731},"OpenAI API Docs: Migrate from prompt objects","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fprompting\u002Fmigrate-from-prompt-object",{"title":733,"url":734},"OpenAI API Docs: Prompt engineering (reusable prompts deprecation note)","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fprompt-engineering",{"title":736,"url":737},"Langfuse Docs: Prompt version control","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Fprompt-version-control",{"title":739,"url":740},"Langfuse Docs: Prompt caching in the SDKs","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Fcaching",{"title":742,"url":743},"Langfuse Docs: Link prompts to traces","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Flink-to-traces",{"title":745,"url":746},"MLflow Docs: Prompt Registry","https:\u002F\u002Fmlflow.org\u002Fdocs\u002Flatest\u002Fgenai\u002Fprompt-registry\u002F",{"title":748,"url":749},"Promptfoo Docs: GitHub Action","https:\u002F\u002Fwww.promptfoo.dev\u002Fdocs\u002Fintegrations\u002Fgithub-action\u002F",{"title":751,"url":752},"Claude Platform Docs: Prompt caching","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fprompt-caching","\u002Fimages\u002Fblog\u002Fprompt-versioning-management\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fprompt-versioning-management\u002Fog.jpg",[60,61,62],"Prompt versioning in production: git, registries, labels and rollbacks","How to version prompts in production: what belongs in a version, git versus a prompt registry, labels and rollbacks, eval gates in CI and cache side effects.","Five upright cards in a row, the third turned back on a curved arrow.",{"slug":760,"published":577,"minutes":761,"category":7,"tags":762,"keywords":767,"about":778,"sources":788,"cover":809,"og":810,"expertise":58,"locales":811,"lang":60,"title":812,"description":813,"coverAlt":814},"llm-batch-api-offline-workloads",9,[763,764,705,765,766],"Batch API","Cost","Prompt caching","Data pipelines",[768,769,770,771,772,773,774,775,776,777],"llm batch api","anthropic message batches api","openai batch api","gemini batch mode","batch inference llm cost","batch api 50 percent discount","openai flex processing","bedrock batch inference","llm offline processing","batch api prompt caching",[779,782,785],{"name":780,"url":781},"Batch processing","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FBatch_processing",{"name":783,"url":784},"JSON Lines","https:\u002F\u002Fjsonlines.org\u002F",{"name":786,"url":787},"Large language model","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FLarge_language_model",[789,792,794,795,798,801,803,806],{"title":790,"url":791},"Claude Platform Docs: Batch processing","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fbatch-processing",{"title":793,"url":665},"Claude Platform Docs: Rate limits (Message Batches API)",{"title":751,"url":752},{"title":796,"url":797},"OpenAI API Docs: Batch API","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fbatch",{"title":799,"url":800},"OpenAI API Docs: Flex processing","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fflex-processing",{"title":802,"url":671},"OpenAI API Docs: Rate limits (batch queue limits)",{"title":804,"url":805},"Google AI for Developers: Gemini Batch API","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fbatch-api",{"title":807,"url":808},"Amazon Bedrock User Guide: Batch inference","https:\u002F\u002Fdocs.aws.amazon.com\u002Fbedrock\u002Flatest\u002Fuserguide\u002Fbatch-inference.html","\u002Fimages\u002Fblog\u002Fllm-batch-api-offline-workloads\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-batch-api-offline-workloads\u002Fog.jpg",[60,61,62],"LLM batch APIs: half-price inference for work that can wait","When to use the Anthropic, OpenAI and Gemini batch APIs, how limits, result matching and failures work, and how batch discounts stack with prompt caching.","A wide box holding eight small squares, tiny squares arriving from the left, one long arrow returning.",1792261156352]