[{"data":1,"prerenderedAt":768},["ShallowReactive",2],{"blog-dspy-en":3},{"slug":4,"published":5,"minutes":6,"category":7,"tags":8,"keywords":14,"about":22,"sources":35,"cover":111,"og":112,"expertise":113,"locales":114,"lang":115,"title":118,"description":119,"coverAlt":120,"metaTitle":121,"takeaways":122,"faq":128,"toc":141,"blocks":169,"others":528},"dspy","2026-10-06",8,"llmops",[9,10,11,12,13],"Prompt optimisation","LLM programs","MIPROv2","GEPA","Python",[4,15,16,17,18,19,20,21],"dspy tutorial","dspy optimizer","miprov2","gepa prompt optimization","dspy vs prompt engineering","automatic prompt optimization","dspy signatures",[23,26,29,32],{"name":24,"url":25},"DSPy","https:\u002F\u002Fdspy.ai",{"name":27,"url":28},"Prompt engineering","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FPrompt_engineering",{"name":30,"url":31},"Bayesian optimization","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FBayesian_optimization",{"name":33,"url":34},"Fine-tuning (deep learning)","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FFine-tuning_(deep_learning)",[36,39,42,45,48,51,54,57,60,63,66,69,72,75,78,81,84,87,90,93,96,99,102,105,108],{"title":37,"url":38},"DSPy home and cost example","https:\u002F\u002Fdspy.ai\u002Fcurrent\u002F",{"title":40,"url":41},"DSPy installation","https:\u002F\u002Fdspy.ai\u002Fcurrent\u002Fgetting-started\u002Finstallation\u002F",{"title":43,"url":44},"DSPy: program, don't prompt","https:\u002F\u002Fdspy.ai\u002Fcurrent\u002Fgetting-started\u002Fprogram-dont-prompt\u002F",{"title":46,"url":47},"DSPy signatures","https:\u002F\u002Fdspy.ai\u002Fcurrent\u002Fdiving-deeper\u002Fsignatures-in-depth\u002F",{"title":49,"url":50},"DSPy metrics","https:\u002F\u002Fdspy.ai\u002Fcurrent\u002Fgetting-started\u002Fmetrics\u002F",{"title":52,"url":53},"DSPy Example API","https:\u002F\u002Fdspy.ai\u002Fcurrent\u002Fapi\u002Fprimitives\u002FExample\u002F",{"title":55,"url":56},"DSPy Predict API","https:\u002F\u002Fdspy.ai\u002Fcurrent\u002Fapi\u002Fmodules\u002FPredict\u002F",{"title":58,"url":59},"DSPy ChainOfThought API","https:\u002F\u002Fdspy.ai\u002Fcurrent\u002Fapi\u002Fmodules\u002FChainOfThought\u002F",{"title":61,"url":62},"DSPy ReAct API","https:\u002F\u002Fdspy.ai\u002Fcurrent\u002Fapi\u002Fmodules\u002FReAct\u002F",{"title":64,"url":65},"DSPy optimizers index","https:\u002F\u002Fdspy.ai\u002Fcurrent\u002Fapi\u002Foptimizers\u002F",{"title":67,"url":68},"DSPy MIPROv2 API","https:\u002F\u002Fdspy.ai\u002Fcurrent\u002Fapi\u002Foptimizers\u002FMIPROv2\u002F",{"title":70,"url":71},"MIPROv2 source code","https:\u002F\u002Fgithub.com\u002Fstanfordnlp\u002Fdspy\u002Fblob\u002Fmain\u002Fdspy\u002Fteleprompt\u002Fmipro_optimizer_v2.py",{"title":73,"url":74},"DSPy BootstrapFewShot API","https:\u002F\u002Fdspy.ai\u002Fcurrent\u002Fapi\u002Foptimizers\u002FBootstrapFewShot\u002F",{"title":76,"url":77},"DSPy BootstrapFinetune API","https:\u002F\u002Fdspy.ai\u002Fcurrent\u002Fapi\u002Foptimizers\u002FBootstrapFinetune\u002F",{"title":79,"url":80},"DSPy GEPA guide","https:\u002F\u002Fdspy.ai\u002Fcurrent\u002Fgetting-started\u002Fgepa-optimization\u002F",{"title":82,"url":83},"GEPA source code","https:\u002F\u002Fgithub.com\u002Fstanfordnlp\u002Fdspy\u002Fblob\u002Fmain\u002Fdspy\u002Fteleprompt\u002Fgepa\u002Fgepa.py",{"title":85,"url":86},"DSPy saving programs","https:\u002F\u002Fdspy.ai\u002Fcurrent\u002Ftutorials\u002Fsaving\u002F",{"title":88,"url":89},"DSPy caching","https:\u002F\u002Fdspy.ai\u002Fcurrent\u002Ftutorials\u002Fcache\u002F",{"title":91,"url":92},"DSPy LM API","https:\u002F\u002Fdspy.ai\u002Fcurrent\u002Fapi\u002Fmodels\u002FLM\u002F",{"title":94,"url":95},"DSPy FAQ","https:\u002F\u002Fdspy.ai\u002Fcurrent\u002Ffaqs\u002F",{"title":97,"url":98},"DSPy GitHub repository","https:\u002F\u002Fgithub.com\u002Fstanfordnlp\u002Fdspy",{"title":100,"url":101},"DSPy 3.4.0 release","https:\u002F\u002Fgithub.com\u002Fstanfordnlp\u002Fdspy\u002Freleases\u002Ftag\u002F3.4.0",{"title":103,"url":104},"DSPy paper, arXiv","https:\u002F\u002Farxiv.org\u002Fabs\u002F2310.03714",{"title":106,"url":107},"MIPRO paper, arXiv","https:\u002F\u002Farxiv.org\u002Fabs\u002F2406.11695",{"title":109,"url":110},"GEPA paper, arXiv","https:\u002F\u002Farxiv.org\u002Fabs\u002F2507.19457","\u002Fimages\u002Fblog\u002Fdspy\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fdspy\u002Fog.jpg","ai-engineer",[115,116,117],"en","de","hu","DSPy review: compile your prompts against a metric, not by hand","DSPy compiles prompts from signatures, a metric and examples. What the optimisers cost in model calls, when they pay off, and when a hand-written prompt wins.","Three small squares dropping through a funnel into an open box, one wider sheet sliding out.","DSPy review: prompts tuned against a metric · Balázs Csorba",[123,124,125,126,127],"DSPy replaces hand-written prompts with signatures, modules and optimisers that search for better prompts against a metric you write.","An optimisation run is paid for in model calls. On MIPROv2's light setting, a one-predictor program makes roughly 750 of them before bootstrapping.","Write the metric first. An optimiser can only improve what the metric measures, and that function is where most of the work sits.","A compiled program is a JSON file of instructions and demonstrations, so it needs version control, and its demonstrations are your training data.","For one prompt on a stable model, a reviewed prompt and an eval suite are simpler. DSPy earns its place in multi-call pipelines that change.",[129,132,135,138],{"q":130,"a":131},"Is DSPy free to use?","The library is MIT-licensed and free. You pay for every model call, including the calls the optimiser makes while it searches. The docs' own example runs cost a few dollars.",{"q":133,"a":134},"How many examples does DSPy need?","I found no fixed minimum in the docs. The FAQ asks for a few example inputs, with labels only when the metric needs them. MIPROv2's light setting uses at most 100 examples for validation.",{"q":136,"a":137},"Can I point DSPy at a local model?","dspy.LM takes LiteLLM-style provider and model strings, so a local endpoint may work if LiteLLM can call it. I did not test one for this review, so check it on your own setup first.",{"q":139,"a":140},"Do I need to re-optimise when I change models?","Yes. The FAQ names a change of target LM as a reason to recompile, because the compiler maps the program onto new prompts. Keep the previous compiled file so you can compare and roll back.",[142,145,148,151,154,157,160,163,166],{"id":143,"title":144},"what-it-is","What it is",{"id":146,"title":147},"how-it-works","How it works",{"id":149,"title":150},"getting-started","Getting started",{"id":152,"title":153},"signatures-and-modules","Signatures and modules",{"id":155,"title":156},"optimisers","Optimisers and what they need",{"id":158,"title":159},"cost-and-deployment","Cost, deployment and data",{"id":161,"title":162},"where-it-falls-short","Where it falls short",{"id":164,"title":165},"verdict","Verdict",{"id":167,"title":168},"sources","Sources",[170,174,177,186,216,217,220,226,227,229,232,233,236,253,256,257,260,306,309,326,333,336,339,342,343,380,383,386,393,399,400,403,406,409,412,413,416,436,449,450],{"type":171,"content":172},"paragraph",[173],"DSPy is an open-source Python framework that treats an LLM pipeline as code. You declare inputs and outputs, pick a module, and let an optimiser search for the instructions and examples that score best on a metric you wrote. The verdict up front: use it for a multi-call pipeline you can measure and expect to change. Skip it for one prompt on a stable model – a reviewed prompt and an eval suite are simpler.",{"type":175,"level":176,"id":143,"text":144},"heading",2,{"type":171,"content":178},[179,180,185],"DSPy calls itself the framework for programming, not prompting, language models. Signatures declare inputs and outputs. Modules decide how the model is asked, from a plain prediction to step-by-step reasoning or a tool loop. Optimisers compile a program against a metric by changing its instructions and examples. It sits between prompt engineering and fine-tuning: the prompt becomes an artefact the optimiser writes. For the wider choice, read ",{"tag":181,"to":182,"children":183},"link","\u002Fblog\u002Ffine-tuning-vs-rag-vs-prompting",[184],"my decision guide on prompting, retrieval and fine-tuning",".",{"type":187,"ordered":188,"items":189},"list",false,[190,201,206,211],[191,195,196,200],{"tag":192,"children":193},"strong",[194],"MIT licence, free to use. ","It installs with ",{"tag":197,"children":198},"code",[199],"pip install dspy"," and needs Python 3.10 or newer.",[202,205],{"tag":192,"children":203},[204],"Latest release 3.4.0, ","published on 25 September. The repository showed 38.6k stars when I checked.",[207,210],{"tag":192,"children":208},[209],"14 optimisers in the API reference, ","from BootstrapFewShot to MIPROv2, GEPA and BootstrapFinetune.",[212,215],{"tag":192,"children":213},[214],"No hosted service that I could find. ","It runs in your process and calls the provider you configure.",{"type":175,"level":176,"id":146,"text":147},{"type":171,"content":218},[219],"A module is built from a signature. At run time the adapter turns the signature into the system message, the model answers, and DSPy parses the output fields. Without an optimiser, the program is only as good as its wording. The optimiser adds a metric, a Python function that scores one prediction, usually from 0.0 to 1.0, and example inputs. It runs the program over the examples, proposes new instructions and demonstrations, keeps the best combination and returns a compiled program.",{"type":221,"src":222,"alt":223,"wide":224,"caption":225},"image","\u002Fimages\u002Fblog\u002Fdspy\u002Fdiagram-6.webp","How DSPy compiles a program A signature and a module form the program. A metric and a set of examples feed the optimiser, which searches instructions and demonstrations and returns a compiled program. The compiled program can be saved as a JSON file and called in the application in place of the original module.",true,"The metric and examples feed the optimiser, which compiles the signature and module into a program you can save.",{"type":175,"level":176,"id":149,"text":150},{"type":197,"code":228},"import os\nimport dspy\n\nlm = dspy.LM(\"openai\u002Fgpt-5-nano\", api_key=os.environ[\"OPENAI_API_KEY\"])\ndspy.configure(lm=lm)\n\n\nclass ExtractIntent(dspy.Signature):\n    \"\"\"Classify the customer's intent in one short email.\"\"\"\n\n    email: str = dspy.InputField()\n    intent: str = dspy.OutputField(desc=\"one of: order, return, invoice, other\")\n\n\nextract = dspy.ChainOfThought(ExtractIntent)\n\n\ndef intent_metric(example, prediction, trace=None):\n    return float(prediction.intent.strip().lower() == example.intent.strip().lower())\n\n\ntrainset = [\n    dspy.Example(email=\"Where is my parcel 4411?\", intent=\"order\").with_inputs(\"email\"),\n    dspy.Example(email=\"I want my money back for the shoes.\", intent=\"return\").with_inputs(\"email\"),\n    # more labelled emails from your own inbox\n]\n\noptimizer = dspy.MIPROv2(metric=intent_metric, auto=\"light\")\ncompiled = optimizer.compile(extract, trainset=trainset)\ncompiled.save(\"intent_v1.json\")",{"type":171,"content":230},[231],"The metric compares predicted and labelled intents, so this example needs labels. The compile call is where the cost sits: light makes hundreds of model calls, so run it on a small set, check the saved file, then scale up.",{"type":175,"level":176,"id":152,"text":153},{"type":171,"content":234},[235],"A signature is the contract. The string form, such as question -> answer, is shorthand. The class form adds a docstring, which becomes the instruction, and typed fields. Field order matters, because reordering inputs or outputs changes the prompt. Test every signature edit as a prompt change.",{"type":187,"ordered":188,"items":237},[238,243,248],[239,242],{"tag":192,"children":240},[241],"dspy.Predict ","maps inputs to outputs with a language model. Its keyword arguments go to that model.",[244,247],{"tag":192,"children":245},[246],"dspy.ChainOfThought ","reasons step by step first. It adds a reasoning field you can customise.",[249,252],{"tag":192,"children":250},[251],"dspy.ReAct ","runs a reason-and-act loop over tools. Its max_iters defaults to 20.",{"type":171,"content":254},[255],"The homepage sums this up as 'same interface, different strategy'. The strategy also sets the bill: reasoning fields add output tokens to every call, and each ReAct step is another model call.",{"type":175,"level":176,"id":155,"text":156},{"type":171,"content":258},[259],"DSPy ships 14 optimisers, from BootstrapFewShot, which collects demonstrations, to GEPA, which rewrites instructions from the metric's feedback. All of them need a metric. The FAQ asks for a task, a metric and a few example inputs, with labels only where the metric needs them. The table covers the four I looked at most closely.",{"type":261,"head":262,"rows":271},"table",[263,265,267,269],[264],"Optimiser",[266],"Changes",[268],"Needs",[270],"Main cost",[272,281,289,297],[273,275,277,279],[274],"BootstrapFewShot",[276],"Few-shot demonstrations",[278],"Metric and training examples",[280],"Program runs, one attempt per example",[282,283,285,287],[11],[284],"Instructions and demonstrations",[286],"Metric and training set",[288],"Trials of 35 examples, plus full validation passes",[290,291,293,295],[12],[292],"Instructions, rewritten by a reflection model",[294],"Metric with feedback and a reflection model",[296],"Budget set by validation size and predictor count",[298,300,302,304],[299],"BootstrapFinetune",[301],"Fine-tuned model per predictor",[303],"Traces and a model you can fine-tune",[305],"One job per model, or per predictor",{"type":171,"content":307},[308],"MIPROv2 is the usual starting point, so its arithmetic matters. Its auto setting fixes the search. For a one-predictor few-shot program, light runs about ten trials and validates on at most 100 examples, medium runs 18 trials on 300 and heavy runs 27 on 1,000. Each trial scores a 35-example minibatch, and a full validation pass runs every sixth trial, at the last trial and for the unoptimised program.",{"type":187,"ordered":188,"items":310},[311,316,321],[312,315],{"tag":192,"children":313},[314],"light: about 750 runs, ","350 for trials and 400 for full passes.",[317,320],{"tag":192,"children":318},[319],"medium: about 2,100 runs, ","630 for trials and 1,500 for full passes.",[322,325],{"tag":192,"children":323},[324],"heavy: about 7,900 runs, ","945 for trials and 7,000 for full passes.",{"type":327,"variant":328,"title":329,"body":330},"callout","note","How to read the counts",[331],[332],"My counts come from the optimiser code, for one predictor with few-shot demonstrations. Bootstrapping and proposal calls come on top, and each extra predictor adds trials and calls. Treat them as orders of magnitude, and count the calls in your first run.",{"type":171,"content":334},[335],"Tokens follow the calls. The FAQ, flagged as possibly out of date for DSPy 2.5 and 2.6, reports about six minutes, 3,200 calls, 2.7 million input tokens and 156,000 output tokens, for about $3 at the OpenAI pricing of the time. That is roughly 850 input and 50 output tokens per call, so the 750 calls of a light run come to about 0.6 million input tokens and 40,000 output tokens. The homepage's current example, GEPA with auto set to medium on 200 examples and gpt-5.4-mini, is listed at $2.18.",{"type":171,"content":337},[338],"GEPA spends its budget differently. Its metric returns a score and feedback text, and a reflection model reads the examples and their scores and proposes rewritten instructions. The guide recommends a larger reflection model than the one you optimise, and the constructor requires one unless you pass a custom proposer. Light targets about six candidate prompts, and the code turns that into a metric-call budget with several full validation passes inside it.",{"type":171,"content":340},[341],"The papers make the strongest case, each on its authors' own tasks. The DSPy paper reports pipelines beating standard few-shot prompting by over 25% and 65% for GPT-3.5 and llama2-13b-chat respectively. The MIPROv2 paper reports wins on five of seven multi-stage programs, with gains up to 13% accuracy. The GEPA paper reports 6% on average over GRPO, a reinforcement-learning baseline, with up to 35 times fewer rollouts.",{"type":175,"level":176,"id":158,"text":159},{"type":261,"head":344,"rows":351},[345,347,349],[346],"Item",[348],"Price",[350],"What it covers",[352,359,366,373],[353,355,357],[354],"DSPy library",[356],"MIT · free",[358],"Installed with pip, runs in your process",[360,362,364],[361],"Model calls at run time",[363],"Your provider's rate",[365],"Every call the program makes, billed per token",[367,369,371],[368],"Vendor example run",[370],"$2.18",[372],"GEPA, auto medium, 200 examples, gpt-5.4-mini",[374,376,378],[375],"Older FAQ run",[377],"About $3",[379],"3,200 calls, 2.7 million input and 156,000 output tokens",{"type":171,"content":381},[382],"Treat the compiled program as a build artefact. Save it as JSON, which the docs call safer and readable, beside the signatures in the same repository, with a version in the file name and the DSPy version pinned. It holds the signature, the demonstrations and the model for each predictor. Loading needs the same program built in code first.",{"type":171,"content":384},[385],"Model changes trigger a recompile, because the compiler maps the program onto new prompts for the new model. Keep the old file, run the metric on both, and promote the new one only if it wins. Pin the version too: the LM page describes an auto engine that prefers a newer backend, so an upgrade can change behaviour.",{"type":171,"content":387},[388,389,185],"The data path is the one you configure, so the processor and region questions match those for any model API. Three things also keep copies of your data. The LM cache is on by default, in memory and on disk. A saved program carries demonstrations drawn from your training data, so its JSON can hold personal data. A GEPA reflection model reads your examples and their scores, which makes it a second processor. For EU options, see ",{"tag":181,"to":390,"children":391},"\u002Fblog\u002Fgdpr-llm-api-eu-data-residency",[392],"my GDPR article on EU data residency",{"type":327,"variant":394,"title":395,"body":396},"warn","Never load a pickle you did not write",[397],[398],"The docs say a pickle can execute arbitrary code, and save_program=True uses cloudpickle with the same risk. Load only the JSON state, from your own repository.",{"type":175,"level":176,"id":161,"text":162},{"type":171,"content":401},[402],"Most tasks do not need an optimiser. The FAQ concedes that for extremely simple settings a plain prompt might work just fine, and you still write the tools, retries and parsing. If you cannot write a metric that matches what a user would call correct, the optimiser improves whatever you did measure, which is not the same thing.",{"type":171,"content":404},[405],"It beats hand-written prompts most clearly when several calls depend on each other, the model changes often, and the output can be scored. Against fine-tuning it is the cheaper and more reversible option for most teams. BootstrapFinetune compiles the program into fine-tuning jobs, but then you serve and version fine-tuned models, one per model or per predictor.",{"type":171,"content":407},[408],"The bill and the metric are the other weak points. A light run makes hundreds of calls, heavy runs thousands, and the validation set drives most of the price. The vendor line that a small, cheap model can often match or beat a hand-prompted frontier one is a hypothesis to test on your own data.",{"type":171,"content":410},[411],"The API is still moving. The 3.4.0 release notes list a breaking change to rlm(...) and remove the old dspy.LMRequest and dspy.LMResponse exports, so read them before each upgrade.",{"type":175,"level":176,"id":164,"text":165},{"type":171,"content":414},[415],"Adopt DSPy when a pipeline is multi-step, measurable and changing. For one prompt on a model you never change, a reviewed prompt and a regression suite do the job. If you cannot write the metric, do not compile anything yet.",{"type":187,"ordered":224,"items":417},[418,423,427,432],[419,422],{"tag":192,"children":420},[421],"Adopt it if ","several model calls must agree and their output can be scored automatically.",[424,426],{"tag":192,"children":425},[421],"the model or the data changes often, since recompiling beats rewriting prompts.",[428,431],{"tag":192,"children":429},[430],"Do not adopt it if ","the task is one prompt on a stable model.",[433,435],{"tag":192,"children":434},[430],"nobody will write and maintain the metric.",{"type":171,"content":437},[438,439,443,444,448],"Three alternatives cover most of the rest. If you want prompts kept in code and changes gated by evals, ",{"tag":181,"to":440,"children":441},"\u002Fblog\u002Fpromptfoo",[442],"Promptfoo"," is the closer fit. If the problem is explicit state and control flow, look at ",{"tag":181,"to":445,"children":446},"\u002Fblog\u002Flanggraph",[447],"LangGraph",", and combine the two if you need both. If the model must learn a format or a style, fine-tuning is the lever, as the decision guide explains.",{"type":175,"level":176,"id":167,"text":168},{"type":187,"ordered":188,"items":451},[452,456,459,462,465,468,471,474,477,480,483,486,489,492,495,498,501,504,507,510,513,516,519,522,525],[453],{"tag":454,"href":38,"children":455},"a",[37],[457],{"tag":454,"href":41,"children":458},[40],[460],{"tag":454,"href":44,"children":461},[43],[463],{"tag":454,"href":47,"children":464},[46],[466],{"tag":454,"href":50,"children":467},[49],[469],{"tag":454,"href":53,"children":470},[52],[472],{"tag":454,"href":56,"children":473},[55],[475],{"tag":454,"href":59,"children":476},[58],[478],{"tag":454,"href":62,"children":479},[61],[481],{"tag":454,"href":65,"children":482},[64],[484],{"tag":454,"href":68,"children":485},[67],[487],{"tag":454,"href":71,"children":488},[70],[490],{"tag":454,"href":74,"children":491},[73],[493],{"tag":454,"href":77,"children":494},[76],[496],{"tag":454,"href":80,"children":497},[79],[499],{"tag":454,"href":83,"children":500},[82],[502],{"tag":454,"href":86,"children":503},[85],[505],{"tag":454,"href":89,"children":506},[88],[508],{"tag":454,"href":92,"children":509},[91],[511],{"tag":454,"href":95,"children":512},[94],[514],{"tag":454,"href":98,"children":515},[97],[517],{"tag":454,"href":101,"children":518},[100],[520],{"tag":454,"href":104,"children":521},[103],[523],{"tag":454,"href":107,"children":524},[106],[526],{"tag":454,"href":110,"children":527},[109],[529,586,654,712],{"slug":530,"published":531,"minutes":532,"category":7,"tags":533,"keywords":539,"about":553,"sources":561,"cover":580,"og":581,"expertise":113,"locales":582,"lang":115,"title":583,"description":584,"coverAlt":585},"deepseek-v4-1-flash-1017","2026-10-17",15,[534,535,536,537,538],"DeepSeek","Open-weight models","KV cache","Mixture of experts","LLM cost",[540,541,542,543,544,545,546,547,548,549,550,551,552],"DeepSeek V4.1 Flash 1017","DeepSeek V4.1 Flash","Causal Encoder-Decoder","Compressed Sparse Attention 2","CSA2","FP4 KV cache","KV cache compression","SWA Bounded Replay","Engram conditional memory","DSpark speculative decoding","DeepSeek pricing","DeepSWE benchmark","open-weight coding model",[554,556,558],{"name":534,"url":555},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FDeepSeek",{"name":537,"url":557},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FMixture_of_experts",{"name":559,"url":560},"Transformer (deep learning architecture)","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FTransformer_(deep_learning_architecture)",[562,565,568,571,574,577],{"title":563,"url":564},"DeepSeek-AI: DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression (arXiv:2609.19969)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.19969",{"title":566,"url":567},"Hugging Face: deepseek-ai\u002FDeepSeek-V4.1-Flash-1017 model card","https:\u002F\u002Fhuggingface.co\u002Fdeepseek-ai\u002FDeepSeek-V4.1-Flash-1017",{"title":569,"url":570},"OpenRouter: DeepSeek V4.1 Flash 1017, pricing and providers","https:\u002F\u002Fopenrouter.ai\u002Fdeepseek\u002Fdeepseek-v4.1-flash-1017",{"title":572,"url":573},"The Decoder: Deepseek V4.1 Flash 1017 beats larger rivals in agentic tasks while once again massively cutting per-task costs","https:\u002F\u002Fthe-decoder.com\u002Fdeepseek-v4-1-flash-1017-beats-larger-rivals-in-agentic-tasks-while-once-again-massively-cutting-per-task-costs\u002F",{"title":575,"url":576},"AI Weekly: DeepSeek opens V4.1-Flash-1017 weights and $0.75 Search API","https:\u002F\u002Faiweekly.co\u002Falerts\u002Fdeepseek-opens-v41-flash-1017-weights-and-075-search-api",{"title":578,"url":579},"DeepSeek on X: DeepSeek-V4.1-Flash-1017 announcement","https:\u002F\u002Fx.com\u002Fdeepseek_ai\u002Fstatus\u002F2111375640717033777","\u002Fimages\u002Fblog\u002Fdeepseek-v4-1-flash-1017\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fdeepseek-v4-1-flash-1017\u002Fog.jpg",[115,116,117],"DeepSeek V4.1 Flash 1017: 890 bytes of KV cache per token, and what that buys an agent","A technical deep dive into DeepSeek V4.1 Flash 1017: the encoder-decoder split, CSA2 cache sharing, FP4 KV cache, the benchmarks and the real cost per agent task.","A tall stack of thin cards on the left, an arrow, and the same cards pressed into a stack a quarter of the height on the right.",{"slug":587,"published":531,"minutes":588,"category":7,"tags":589,"keywords":595,"about":607,"sources":617,"cover":648,"og":649,"expertise":113,"locales":650,"lang":115,"title":651,"description":652,"coverAlt":653},"llm-rate-limits-retries-provider-fallback",10,[590,591,592,593,594],"Rate limits","Retries","Fallback","Reliability","LLM gateway",[596,597,598,599,600,601,602,603,604,605,606],"llm rate limit handling","429 too many requests llm api","exponential backoff with jitter","anthropic 529 overloaded error","openai rate limit retry","llm provider fallback","litellm fallbacks","retry-after header","llm circuit breaker","multi provider llm failover","claude api rate limits",[608,611,614],{"name":609,"url":610},"Token bucket","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FToken_bucket",{"name":612,"url":613},"Exponential backoff","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FExponential_backoff",{"name":615,"url":616},"Circuit breaker design pattern","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FCircuit_breaker_design_pattern",[618,621,624,627,630,633,636,639,642,645],{"title":619,"url":620},"Claude Platform Docs: Rate limits","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fapi\u002Frate-limits",{"title":622,"url":623},"Claude Platform Docs: API errors","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fapi\u002Ferrors",{"title":625,"url":626},"OpenAI API Docs: Rate limits","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Frate-limits",{"title":628,"url":629},"AWS Builders Library: Timeouts, retries and backoff with jitter","https:\u002F\u002Fbuilder.aws.com\u002Fcontent\u002F3EumjoZascWd1oZiEgL8ORlv3qE\u002Ftimeouts-retries-and-backoff-with-jitter",{"title":631,"url":632},"MDN: Retry-After header","https:\u002F\u002Fdeveloper.mozilla.org\u002Fen-US\u002Fdocs\u002FWeb\u002FHTTP\u002FReference\u002FHeaders\u002FRetry-After",{"title":634,"url":635},"Martin Fowler: Circuit Breaker (2014)","https:\u002F\u002Fmartinfowler.com\u002Fbliki\u002FCircuitBreaker.html",{"title":637,"url":638},"LiteLLM Docs: Router, retries and cooldowns","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Frouting",{"title":640,"url":641},"LiteLLM Docs: Proxy fallbacks and reliability","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Fproxy\u002Freliability",{"title":643,"url":644},"OpenRouter Docs: Model fallbacks","https:\u002F\u002Fopenrouter.ai\u002Fdocs\u002Fguides\u002Frouting\u002Fmodel-fallbacks",{"title":646,"url":647},"AI SDK Docs: Settings (maxRetries, timeout)","https:\u002F\u002Fai-sdk.dev\u002Fdocs\u002Fai-sdk-core\u002Fsettings","\u002Fimages\u002Fblog\u002Fllm-rate-limits-retries-provider-fallback\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-rate-limits-retries-provider-fallback\u002Fog.jpg",[115,116,117],"LLM rate limits in production: retries, backoff and provider fallback","How LLM rate limits work, which errors to retry, backoff with jitter, retrying in one layer only, and fallback chains that do not break your output.","Small squares climbing three steps up to a wide rectangle, two more rectangles behind it in a row.",{"slug":655,"published":531,"minutes":6,"category":7,"tags":656,"keywords":662,"about":673,"sources":681,"cover":706,"og":707,"expertise":113,"locales":708,"lang":115,"title":709,"description":710,"coverAlt":711},"prompt-versioning-management",[657,658,659,660,661],"Prompt management","Versioning","LLMOps","Evals","Langfuse",[663,664,665,666,667,668,669,670,671,672],"prompt versioning","prompt management","prompt registry","version control for prompts","langfuse prompt management","mlflow prompt registry","prompts in git","openai prompt object deprecation","prompt rollback production","prompt ci evaluation",[674,675,678],{"name":27,"url":28},{"name":676,"url":677},"Version control","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FVersion_control",{"name":679,"url":680},"Continuous integration","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FContinuous_integration",[682,685,688,691,694,697,700,703],{"title":683,"url":684},"OpenAI API Docs: Migrate from prompt objects","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fprompting\u002Fmigrate-from-prompt-object",{"title":686,"url":687},"OpenAI API Docs: Prompt engineering (reusable prompts deprecation note)","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fprompt-engineering",{"title":689,"url":690},"Langfuse Docs: Prompt version control","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Fprompt-version-control",{"title":692,"url":693},"Langfuse Docs: Prompt caching in the SDKs","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Fcaching",{"title":695,"url":696},"Langfuse Docs: Link prompts to traces","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Flink-to-traces",{"title":698,"url":699},"MLflow Docs: Prompt Registry","https:\u002F\u002Fmlflow.org\u002Fdocs\u002Flatest\u002Fgenai\u002Fprompt-registry\u002F",{"title":701,"url":702},"Promptfoo Docs: GitHub Action","https:\u002F\u002Fwww.promptfoo.dev\u002Fdocs\u002Fintegrations\u002Fgithub-action\u002F",{"title":704,"url":705},"Claude Platform Docs: Prompt caching","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fprompt-caching","\u002Fimages\u002Fblog\u002Fprompt-versioning-management\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fprompt-versioning-management\u002Fog.jpg",[115,116,117],"Prompt versioning in production: git, registries, labels and rollbacks","How to version prompts in production: what belongs in a version, git versus a prompt registry, labels and rollbacks, eval gates in CI and cache side effects.","Five upright cards in a row, the third turned back on a curved arrow.",{"slug":713,"published":531,"minutes":714,"category":7,"tags":715,"keywords":720,"about":731,"sources":741,"cover":762,"og":763,"expertise":113,"locales":764,"lang":115,"title":765,"description":766,"coverAlt":767},"llm-batch-api-offline-workloads",9,[716,717,659,718,719],"Batch API","Cost","Prompt caching","Data pipelines",[721,722,723,724,725,726,727,728,729,730],"llm batch api","anthropic message batches api","openai batch api","gemini batch mode","batch inference llm cost","batch api 50 percent discount","openai flex processing","bedrock batch inference","llm offline processing","batch api prompt caching",[732,735,738],{"name":733,"url":734},"Batch processing","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FBatch_processing",{"name":736,"url":737},"JSON Lines","https:\u002F\u002Fjsonlines.org\u002F",{"name":739,"url":740},"Large language model","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FLarge_language_model",[742,745,747,748,751,754,756,759],{"title":743,"url":744},"Claude Platform Docs: Batch processing","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fbatch-processing",{"title":746,"url":620},"Claude Platform Docs: Rate limits (Message Batches API)",{"title":704,"url":705},{"title":749,"url":750},"OpenAI API Docs: Batch API","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fbatch",{"title":752,"url":753},"OpenAI API Docs: Flex processing","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fflex-processing",{"title":755,"url":626},"OpenAI API Docs: Rate limits (batch queue limits)",{"title":757,"url":758},"Google AI for Developers: Gemini Batch API","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fbatch-api",{"title":760,"url":761},"Amazon Bedrock User Guide: Batch inference","https:\u002F\u002Fdocs.aws.amazon.com\u002Fbedrock\u002Flatest\u002Fuserguide\u002Fbatch-inference.html","\u002Fimages\u002Fblog\u002Fllm-batch-api-offline-workloads\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-batch-api-offline-workloads\u002Fog.jpg",[115,116,117],"LLM batch APIs: half-price inference for work that can wait","When to use the Anthropic, OpenAI and Gemini batch APIs, how limits, result matching and failures work, and how batch discounts stack with prompt caching.","A wide box holding eight small squares, tiny squares arriving from the left, one long arrow returning.",1792261155628]