[{"data":1,"prerenderedAt":606},["ShallowReactive",2],{"blog-promptfoo-en":3},{"slug":4,"published":5,"minutes":6,"category":7,"tags":8,"keywords":13,"about":21,"sources":25,"cover":50,"og":51,"expertise":52,"locales":53,"lang":54,"title":57,"description":58,"coverAlt":59,"metaTitle":60,"takeaways":61,"faq":67,"toc":80,"blocks":104,"others":364},"promptfoo","2026-07-07",10,"llmops",[9,10,11,12],"LLM evaluation","Red teaming","Prompt testing","CI\u002FCD",[4,14,15,16,17,18,19,20],"promptfoo vs langsmith","LLM evaluation framework","promptfoo red teaming","promptfooconfig.yaml example","LLM evals in CI","promptfoo pricing","open source LLM testing",[22],{"name":23,"url":24},"Promptfoo","https:\u002F\u002Fwww.promptfoo.dev",[26,29,32,35,38,41,44,47],{"title":27,"url":28},"Promptfoo documentation: intro","https:\u002F\u002Fwww.promptfoo.dev\u002Fdocs\u002Fintro\u002F",{"title":30,"url":31},"Promptfoo pricing: plan comparison","https:\u002F\u002Fwww.promptfoo.dev\u002Fpricing",{"title":33,"url":34},"Promptfoo documentation: getting started","https:\u002F\u002Fwww.promptfoo.dev\u002Fdocs\u002Fgetting-started\u002F",{"title":36,"url":37},"Promptfoo documentation: command line usage (exit codes)","https:\u002F\u002Fwww.promptfoo.dev\u002Fdocs\u002Fusage\u002Fcommand-line\u002F",{"title":39,"url":40},"Promptfoo documentation: red team plugins","https:\u002F\u002Fwww.promptfoo.dev\u002Fdocs\u002Fred-team\u002Fplugins\u002F",{"title":42,"url":43},"Promptfoo documentation: assertions and metrics","https:\u002F\u002Fwww.promptfoo.dev\u002Fdocs\u002Fconfiguration\u002Fexpected-outputs\u002F",{"title":45,"url":46},"Promptfoo repository on GitHub (MIT licence)","https:\u002F\u002Fgithub.com\u002Fpromptfoo\u002Fpromptfoo",{"title":48,"url":49},"Promptfoo blog: Promptfoo is joining OpenAI (9 March 2026)","https:\u002F\u002Fwww.promptfoo.dev\u002Fblog\u002Fpromptfoo-joining-openai\u002F","\u002Fimages\u002Fblog\u002Fpromptfoo\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fpromptfoo\u002Fog.jpg","ai-engineer",[54,55,56],"en","de","hu","Promptfoo: LLM evals and red teaming from one YAML file","Promptfoo in 2026: MIT-licensed evals and red teaming, now inside OpenAI. What it does well, where the YAML approach breaks, and what the tiers cost.","A tall sheet inside a wider frame, five arrows fanning to five squares, two under a shield.","Promptfoo review: evals and red teaming · Balázs Csorba",[62,63,64,65,66],"Promptfoo is a MIT-licensed CLI that runs quality evals and red teaming from one YAML file, and the repository declared itself part of OpenAI after the March 2026 acquisition announcement.","The Community tier is free and includes 10,000 red team probes a month; dashboards, custom plugins, SSO and the API are quote-only Enterprise rows.","Declarative test cases are the real strength, because a reviewer can read an eval change as a diff without running a Python test suite.","Model-graded assertions such as llm-rubric are nondeterministic and cost a judge call per cell, so the pass rate measures the grader as much as the prompt.","Keep the config and the test data portable: MIT keeps the code forkable, but a model-comparison tool owned by one candidate is a neutrality problem the licence does not solve.",[68,71,74,77],{"q":69,"a":70},"Is promptfoo free to use?","Yes. The Community version is MIT-licensed, runs locally and includes all evaluation features plus up to 10,000 red team probes per month. Enterprise adds custom probe limits, team sharing, continuous monitoring, SSO and API access, and both Enterprise and On-Premise are priced on request.",{"q":72,"a":73},"What is the difference between promptfoo evals and red teaming?","Evals run your prompts against test cases and grade the outputs with assertions such as substring checks, cost thresholds or llm-rubric. Red teaming generates adversarial payloads against a configured target: promptfoo redteam run executes the probes and promptfoo redteam report turns the findings into a vulnerability report.",{"q":75,"a":76},"Which model providers does promptfoo support?","The documentation lists more than 60 providers, including OpenAI, Anthropic, Google, Azure, Bedrock and local models through Ollama. Anything else can be reached through a custom HTTP, Python or JavaScript provider, so the config file does not have to change when the model behind an endpoint does.",{"q":78,"a":79},"What happened to promptfoo after the OpenAI acquisition?","Promptfoo announced on 9 March 2026 that it had agreed to be acquired by OpenAI, stating that the product would remain open source and MIT licensed and that the team would continue to serve customers. The repository now describes promptfoo as part of OpenAI, and the documentation is still being updated.",[81,84,87,90,92,95,98,101],{"id":82,"title":83},"what-it-is","What it is",{"id":85,"title":86},"how-it-works","How it works",{"id":88,"title":89},"getting-started","Getting started",{"id":91,"title":10},"red-teaming",{"id":93,"title":94},"ownership","Who owns it now",{"id":96,"title":97},"where-it-shingles","Where it shingles",{"id":99,"title":100},"verdict","Verdict",{"id":102,"title":103},"sources","Sources",[105,109,112,115,123,149,152,153,160,166,172,173,176,178,187,194,195,213,223,227,247,250,251,254,257,258,261,307,310,316,317,320,332,336,337],{"type":106,"content":107},"paragraph",[108],"Promptfoo is an open-source CLI and library that runs LLM evaluations and adversarial red teaming from a single YAML file. Position up front: it is the most practical way to turn a prompt change or a model swap into a text diff that CI can fail on, and since March 2026 it is also an eval tool that belongs to OpenAI.",{"type":106,"content":110},[111],"It sits between a folder of pytest scripts and a hosted evaluation platform. Compared with LangSmith and Braintrust it gives up dashboards, shared history and a team UI in the free tier, and in exchange it runs on your machine, talks directly to more than 60 providers and needs no account. Compared with DeepEval it gives up Python expressiveness for a config file that a non-programmer can read in review.",{"type":113,"level":114,"id":82,"text":83},"heading",2,{"type":106,"content":116},[117,118,122],"Two products share one repository and one config format. The evaluation side runs prompts against test cases and grades the outputs; the red teaming side attacks a configured target with generated payloads and files the results as a vulnerability report. Both are driven from ",{"tag":119,"children":120},"code",[121],"promptfooconfig.yaml",", both run from the same command line and both can be wired into CI.",{"type":124,"ordered":125,"items":126},"list",false,[127,129,131,133,143,145,147],[128],"MIT-licensed CLI and library: version 0.124.0 on npm, about 25.8k stars and 2.4k forks on GitHub",[130],"Declarative configuration: prompts, providers, tests and assertions in one YAML file",[132],"More than 60 providers, including OpenAI, Anthropic, Google, Azure, Bedrock, Ollama and custom HTTP, Python or JavaScript endpoints",[134,135,138,139,142],"Two commands cover most of the work: ",{"tag":119,"children":136},[137],"promptfoo eval"," for quality, ",{"tag":119,"children":140},[141],"promptfoo redteam run"," for security",[144],"The Community tier is free and includes 10,000 red team probes a month; Enterprise and On-Premise are priced on request",[146],"The vendor reports 350k developers, 130k monthly actives and more than 25% of the Fortune 500",[148],"The site footer carries SOC 2 and ISO 27001 badges, and the documentation ships a GitHub Action for CI",{"type":106,"content":150},[151],"What the free tier deliberately withholds is the hosted part. There is no team dashboard, no searchable scan history, no continuous monitoring and no API in Community; those are Enterprise rows in the feature comparison. The tool itself stays a local process that calls your providers and writes results to your disk.",{"type":113,"level":114,"id":85,"text":86},{"type":106,"content":154},[155,156,159],"An evaluation is a matrix. The config declares prompts, providers and test cases, the runner expands the cross product, sends every cell to every model and scores the response against the assertions attached to that cell. Assertions are typed: substring and JSON schema checks, similarity, cost and latency thresholds, and model-graded checks such as ",{"tag":119,"children":157},[158],"llm-rubric"," that spend an extra model call to judge the output.",{"type":161,"src":162,"alt":163,"wide":164,"caption":165},"image","\u002Fimages\u002Fblog\u002Fpromptfoo\u002Fdiagram-8.webp","How one promptfoo evaluation runs A four stage pipeline: the config file feeds a runner that calls every provider in parallel with caching, assertions grade each output, and the results go to a web viewer and a CI exit code. Below, a matrix of prompts against providers with pass and fail cells.",true,"The promptfoo evaluation matrix: every prompt runs against every provider, assertions grade each cell, and the exit code is what CI reacts to.",{"type":106,"content":167},[168,169,171],"The runner is concurrent and caches completed calls, so a rerun after a small edit mostly re-reads the cache. Results open in a local web viewer as a side-by-side matrix, and the CLI can write JSON, YAML, CSV or HTML. The exit codes are the integration point: ",{"tag":119,"children":170},[137]," returns 100 when at least one test fails or the pass rate falls below PROMPTFOO_PASS_RATE_THRESHOLD, and 1 for any other error.",{"type":113,"level":114,"id":88,"text":89},{"type":106,"content":174},[175],"Install it with npm, brew or pip, or skip the install with npx promptfoo@latest. A minimal config that compares two models on one translated sentence:",{"type":119,"code":177},"# promptfooconfig.yaml: two models, one prompt, three checks\nprompts:\n  - 'Translate this sentence into {{language}}: {{input}}'\n\nproviders:\n  - openai:gpt-6-sol\n  - openai:gpt-6-luna\n\ntests:\n  - vars:\n      language: French\n      input: Hello world\n    assert:\n      - type: contains\n        value: 'Bonjour'\n      - type: llm-rubric\n        value: 'the translation is idiomatic and complete'\n      - type: cost\n        threshold: 0.01",{"type":106,"content":179},[180,182,183,186],{"tag":119,"children":181},[137]," runs every cell of that matrix and prints a pass rate; ",{"tag":119,"children":184},[185],"promptfoo view"," opens the side-by-side comparison in a browser. The -r flag swaps providers from the command line without editing the file, which is how the documentation suggests checking a candidate model against a baseline.",{"type":188,"variant":189,"title":190,"body":191},"callout","tip","Cache before you iterate",[192],[193],"The runner hashes the config, the variables and the provider parameters, so an unchanged cell is not sent to the API again on a rerun. Model-graded assertions are the exception: they are judged at run time, so every one of them costs a judge call per cell.",{"type":113,"level":114,"id":91,"text":10},{"type":106,"content":196},[197,200,201,204,205,208,209,212],{"tag":119,"children":198},[199],"promptfoo redteam init"," writes a red team config, ",{"tag":119,"children":202},[203],"redteam setup"," walks through the target, ",{"tag":119,"children":206},[207],"redteam run"," generates and executes the attacks, and ",{"tag":119,"children":210},[211],"redteam report"," renders the findings. The documentation describes each plugin as a trained model that produces payloads for a specific weakness, rather than a fixed wordlist.",{"type":124,"ordered":125,"items":214},[215,217,219,221],[216],"157 plugins across six categories: brand, compliance and legal, dataset, security and access control, trust and safety, and custom",[218],"Security plugins map to the OWASP Top 10 for LLMs, the OWASP API Security Top 10 and MITRE ATLAS",[220],"Compliance plugins map to NIST AI RMF, ISO\u002FIEC 42001, GDPR Article 5 and EU AI Act Article 5",[222],"Plugins are selected per target in the config, so a scan can be narrowed to the categories one application actually exposes",{"type":113,"level":224,"id":225,"text":226},3,"framework-mappings","Framework mappings",{"type":106,"content":228},[229,230,233,234,233,237,233,240,233,243,246],"For a team that has to show coverage rather than a pile of findings, the framework IDs are the useful part: ",{"tag":119,"children":231},[232],"nist:ai:measure:1.1",", ",{"tag":119,"children":235},[236],"owasp:llm:01",{"tag":119,"children":238},[239],"iso:42001:privacy",{"tag":119,"children":241},[242],"gdpr:art5",{"tag":119,"children":244},[245],"eu:ai-act:art5",". A report grouped by those IDs is a coverage argument a reviewer can follow. This is compliance mechanics rather than a compliance claim: the tool shows which probes ran, not that the product satisfies the regulation.",{"type":106,"content":248},[249],"The free ceiling is 10,000 probes a month, and the plugins that need inference to generate and grade payloads are what consume it. Past that, the pricing page points at Enterprise for custom limits.",{"type":113,"level":114,"id":93,"text":94},{"type":106,"content":252},[253],"On 9 March 2026 Promptfoo announced that it had agreed to be acquired by OpenAI. The post promised that the product would remain open source and MIT licensed and that the team would keep serving customers, and it noted that closing was subject to customary closing conditions. The repository now carries the line that Promptfoo is part of OpenAI, and the documentation was still being updated on 7 October 2026.",{"type":106,"content":255},[256],"The licence keeps the code forkable; it does not keep the roadmap neutral. The engineering worry is narrow and concrete: a tool whose job is to answer which model should we ship is now owned by one of the candidates, and its red teaming, guardrails and model security products sit next to OpenAI's agent platform. Nothing in the MIT licence prevents a fork, and nothing in it obliges the company to prioritise multi-model comparison either.",{"type":113,"level":114,"id":96,"text":97},{"type":106,"content":259},[260],"The first weakness is the config itself. A file with a dozen prompts and forty test cases is readable; one with four hundred cases is not, and anything that needs a loop, a join against production data or a custom parser pushes you into JavaScript assertion callbacks, at which point the language you avoided is back. The second is grading drift: model-graded assertions move between judge versions, so a green suite can turn amber because the grader moved. The third is scope, because a security team that needs an audit trail buys Enterprise regardless of how good the evals are.",{"type":262,"head":263,"rows":272},"table",[264,266,268,270],[265],"Tool",[267],"Interface",[269],"Free tier",[271],"Cost model",[273,281,290,298],[274,275,277,279],[4],[276],"YAML and CLI, runs locally",[278],"Full evals, 10k red team probes a month",[280],"MIT; Enterprise quoted",[282,284,286,288],[283],"LangSmith",[285],"Hosted platform plus SDK",[287],"1 seat and 5k base traces a month",[289],"Plus from 39 USD per seat a month",[291,293,294,296],[292],"Braintrust",[285],[295],"Unlimited users, 1 GB processed data",[297],"Pro at 249 USD a month",[299,301,303,305],[300],"DeepEval",[302],"Python, pytest style",[304],"The whole framework",[306],"Apache 2.0; hosted platform behind it",{"type":106,"content":308},[309],"The commercial shape is a free harness with a paid control plane. Continuous monitoring, a central dashboard, custom plugins, organisation-wide attack profiles, SSO, saved targets, searchable history and the API are Enterprise rows, and on-premise deployment adds a second quote. A solo developer never needs them; a team that has to prove what it tested last quarter does.",{"type":188,"variant":311,"title":312,"body":313},"warn","Grader confidence",[314],[315],"Model-graded assertions report a score, not a measurement. Pin the judge model, keep a small human-labelled sample in the suite and re-run it whenever the judge changes; otherwise the pass rate measures the grader as much as the prompt.",{"type":113,"level":114,"id":99,"text":100},{"type":106,"content":318},[319],"Promptfoo is a good local harness that has grown a serious security half. It is fast, reviewable and honest about what it measures, and the red teaming coverage is now specific enough to argue with. The open question is not the licence, which stays MIT, but who decides which models the tool is tuned to notice.",{"type":124,"ordered":164,"items":321},[322,324,326,328,330],[323],"Take it if you want prompt and model changes gated in CI as text diffs, with an exit code that fails the job.",[325],"Take it if you need adversarial testing before release and want it reviewed in the same pull request as your quality evals.",[327],"Skip it if you need a hosted service, because dashboards, shared history, scan search and the API are Enterprise-only.",[329],"Skip it if your evaluation logic needs real code, because a Python framework costs less effort than a YAML file fighting to express a loop.",[331],"Keep your test data portable: the config is MIT and forkable, but a model-comparison tool owned by one of the candidates is a neutrality problem the licence does not solve.",{"type":333,"content":334},"quote",[335],"A suite you can read in a pull request beats a dashboard you have to click through, and promptfoo is built on that premise. Its new owner is the reason to keep the exit code and the test data portable.",{"type":113,"level":114,"id":102,"text":103},{"type":124,"ordered":164,"items":338},[339,343,346,349,352,355,358,361],[340],{"tag":341,"href":28,"children":342},"a",[27],[344],{"tag":341,"href":31,"children":345},[30],[347],{"tag":341,"href":34,"children":348},[33],[350],{"tag":341,"href":37,"children":351},[36],[353],{"tag":341,"href":40,"children":354},[39],[356],{"tag":341,"href":43,"children":357},[42],[359],{"tag":341,"href":46,"children":360},[45],[362],{"tag":341,"href":49,"children":363},[48],[365,422,489,550],{"slug":366,"published":367,"minutes":368,"category":7,"tags":369,"keywords":375,"about":389,"sources":397,"cover":416,"og":417,"expertise":52,"locales":418,"lang":54,"title":419,"description":420,"coverAlt":421},"deepseek-v4-1-flash-1017","2026-10-17",15,[370,371,372,373,374],"DeepSeek","Open-weight models","KV cache","Mixture of experts","LLM cost",[376,377,378,379,380,381,382,383,384,385,386,387,388],"DeepSeek V4.1 Flash 1017","DeepSeek V4.1 Flash","Causal Encoder-Decoder","Compressed Sparse Attention 2","CSA2","FP4 KV cache","KV cache compression","SWA Bounded Replay","Engram conditional memory","DSpark speculative decoding","DeepSeek pricing","DeepSWE benchmark","open-weight coding model",[390,392,394],{"name":370,"url":391},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FDeepSeek",{"name":373,"url":393},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FMixture_of_experts",{"name":395,"url":396},"Transformer (deep learning architecture)","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FTransformer_(deep_learning_architecture)",[398,401,404,407,410,413],{"title":399,"url":400},"DeepSeek-AI: DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression (arXiv:2609.19969)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.19969",{"title":402,"url":403},"Hugging Face: deepseek-ai\u002FDeepSeek-V4.1-Flash-1017 model card","https:\u002F\u002Fhuggingface.co\u002Fdeepseek-ai\u002FDeepSeek-V4.1-Flash-1017",{"title":405,"url":406},"OpenRouter: DeepSeek V4.1 Flash 1017, pricing and providers","https:\u002F\u002Fopenrouter.ai\u002Fdeepseek\u002Fdeepseek-v4.1-flash-1017",{"title":408,"url":409},"The Decoder: Deepseek V4.1 Flash 1017 beats larger rivals in agentic tasks while once again massively cutting per-task costs","https:\u002F\u002Fthe-decoder.com\u002Fdeepseek-v4-1-flash-1017-beats-larger-rivals-in-agentic-tasks-while-once-again-massively-cutting-per-task-costs\u002F",{"title":411,"url":412},"AI Weekly: DeepSeek opens V4.1-Flash-1017 weights and $0.75 Search API","https:\u002F\u002Faiweekly.co\u002Falerts\u002Fdeepseek-opens-v41-flash-1017-weights-and-075-search-api",{"title":414,"url":415},"DeepSeek on X: DeepSeek-V4.1-Flash-1017 announcement","https:\u002F\u002Fx.com\u002Fdeepseek_ai\u002Fstatus\u002F2111375640717033777","\u002Fimages\u002Fblog\u002Fdeepseek-v4-1-flash-1017\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fdeepseek-v4-1-flash-1017\u002Fog.jpg",[54,55,56],"DeepSeek V4.1 Flash 1017: 890 bytes of KV cache per token, and what that buys an agent","A technical deep dive into DeepSeek V4.1 Flash 1017: the encoder-decoder split, CSA2 cache sharing, FP4 KV cache, the benchmarks and the real cost per agent task.","A tall stack of thin cards on the left, an arrow, and the same cards pressed into a stack a quarter of the height on the right.",{"slug":423,"published":367,"minutes":6,"category":7,"tags":424,"keywords":430,"about":442,"sources":452,"cover":483,"og":484,"expertise":52,"locales":485,"lang":54,"title":486,"description":487,"coverAlt":488},"llm-rate-limits-retries-provider-fallback",[425,426,427,428,429],"Rate limits","Retries","Fallback","Reliability","LLM gateway",[431,432,433,434,435,436,437,438,439,440,441],"llm rate limit handling","429 too many requests llm api","exponential backoff with jitter","anthropic 529 overloaded error","openai rate limit retry","llm provider fallback","litellm fallbacks","retry-after header","llm circuit breaker","multi provider llm failover","claude api rate limits",[443,446,449],{"name":444,"url":445},"Token bucket","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FToken_bucket",{"name":447,"url":448},"Exponential backoff","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FExponential_backoff",{"name":450,"url":451},"Circuit breaker design pattern","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FCircuit_breaker_design_pattern",[453,456,459,462,465,468,471,474,477,480],{"title":454,"url":455},"Claude Platform Docs: Rate limits","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fapi\u002Frate-limits",{"title":457,"url":458},"Claude Platform Docs: API errors","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fapi\u002Ferrors",{"title":460,"url":461},"OpenAI API Docs: Rate limits","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Frate-limits",{"title":463,"url":464},"AWS Builders Library: Timeouts, retries and backoff with jitter","https:\u002F\u002Fbuilder.aws.com\u002Fcontent\u002F3EumjoZascWd1oZiEgL8ORlv3qE\u002Ftimeouts-retries-and-backoff-with-jitter",{"title":466,"url":467},"MDN: Retry-After header","https:\u002F\u002Fdeveloper.mozilla.org\u002Fen-US\u002Fdocs\u002FWeb\u002FHTTP\u002FReference\u002FHeaders\u002FRetry-After",{"title":469,"url":470},"Martin Fowler: Circuit Breaker (2014)","https:\u002F\u002Fmartinfowler.com\u002Fbliki\u002FCircuitBreaker.html",{"title":472,"url":473},"LiteLLM Docs: Router, retries and cooldowns","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Frouting",{"title":475,"url":476},"LiteLLM Docs: Proxy fallbacks and reliability","https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Fproxy\u002Freliability",{"title":478,"url":479},"OpenRouter Docs: Model fallbacks","https:\u002F\u002Fopenrouter.ai\u002Fdocs\u002Fguides\u002Frouting\u002Fmodel-fallbacks",{"title":481,"url":482},"AI SDK Docs: Settings (maxRetries, timeout)","https:\u002F\u002Fai-sdk.dev\u002Fdocs\u002Fai-sdk-core\u002Fsettings","\u002Fimages\u002Fblog\u002Fllm-rate-limits-retries-provider-fallback\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-rate-limits-retries-provider-fallback\u002Fog.jpg",[54,55,56],"LLM rate limits in production: retries, backoff and provider fallback","How LLM rate limits work, which errors to retry, backoff with jitter, retrying in one layer only, and fallback chains that do not break your output.","Small squares climbing three steps up to a wide rectangle, two more rectangles behind it in a row.",{"slug":490,"published":367,"minutes":491,"category":7,"tags":492,"keywords":498,"about":509,"sources":519,"cover":544,"og":545,"expertise":52,"locales":546,"lang":54,"title":547,"description":548,"coverAlt":549},"prompt-versioning-management",8,[493,494,495,496,497],"Prompt management","Versioning","LLMOps","Evals","Langfuse",[499,500,501,502,503,504,505,506,507,508],"prompt versioning","prompt management","prompt registry","version control for prompts","langfuse prompt management","mlflow prompt registry","prompts in git","openai prompt object deprecation","prompt rollback production","prompt ci evaluation",[510,513,516],{"name":511,"url":512},"Prompt engineering","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FPrompt_engineering",{"name":514,"url":515},"Version control","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FVersion_control",{"name":517,"url":518},"Continuous integration","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FContinuous_integration",[520,523,526,529,532,535,538,541],{"title":521,"url":522},"OpenAI API Docs: Migrate from prompt objects","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fprompting\u002Fmigrate-from-prompt-object",{"title":524,"url":525},"OpenAI API Docs: Prompt engineering (reusable prompts deprecation note)","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fprompt-engineering",{"title":527,"url":528},"Langfuse Docs: Prompt version control","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Fprompt-version-control",{"title":530,"url":531},"Langfuse Docs: Prompt caching in the SDKs","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Fcaching",{"title":533,"url":534},"Langfuse Docs: Link prompts to traces","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fprompt-management\u002Ffeatures\u002Flink-to-traces",{"title":536,"url":537},"MLflow Docs: Prompt Registry","https:\u002F\u002Fmlflow.org\u002Fdocs\u002Flatest\u002Fgenai\u002Fprompt-registry\u002F",{"title":539,"url":540},"Promptfoo Docs: GitHub Action","https:\u002F\u002Fwww.promptfoo.dev\u002Fdocs\u002Fintegrations\u002Fgithub-action\u002F",{"title":542,"url":543},"Claude Platform Docs: Prompt caching","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fprompt-caching","\u002Fimages\u002Fblog\u002Fprompt-versioning-management\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fprompt-versioning-management\u002Fog.jpg",[54,55,56],"Prompt versioning in production: git, registries, labels and rollbacks","How to version prompts in production: what belongs in a version, git versus a prompt registry, labels and rollbacks, eval gates in CI and cache side effects.","Five upright cards in a row, the third turned back on a curved arrow.",{"slug":551,"published":367,"minutes":552,"category":7,"tags":553,"keywords":558,"about":569,"sources":579,"cover":600,"og":601,"expertise":52,"locales":602,"lang":54,"title":603,"description":604,"coverAlt":605},"llm-batch-api-offline-workloads",9,[554,555,495,556,557],"Batch API","Cost","Prompt caching","Data pipelines",[559,560,561,562,563,564,565,566,567,568],"llm batch api","anthropic message batches api","openai batch api","gemini batch mode","batch inference llm cost","batch api 50 percent discount","openai flex processing","bedrock batch inference","llm offline processing","batch api prompt caching",[570,573,576],{"name":571,"url":572},"Batch processing","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FBatch_processing",{"name":574,"url":575},"JSON Lines","https:\u002F\u002Fjsonlines.org\u002F",{"name":577,"url":578},"Large language model","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FLarge_language_model",[580,583,585,586,589,592,594,597],{"title":581,"url":582},"Claude Platform Docs: Batch processing","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fbuild-with-claude\u002Fbatch-processing",{"title":584,"url":455},"Claude Platform Docs: Rate limits (Message Batches API)",{"title":542,"url":543},{"title":587,"url":588},"OpenAI API Docs: Batch API","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fbatch",{"title":590,"url":591},"OpenAI API Docs: Flex processing","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fflex-processing",{"title":593,"url":461},"OpenAI API Docs: Rate limits (batch queue limits)",{"title":595,"url":596},"Google AI for Developers: Gemini Batch API","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fbatch-api",{"title":598,"url":599},"Amazon Bedrock User Guide: Batch inference","https:\u002F\u002Fdocs.aws.amazon.com\u002Fbedrock\u002Flatest\u002Fuserguide\u002Fbatch-inference.html","\u002Fimages\u002Fblog\u002Fllm-batch-api-offline-workloads\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fllm-batch-api-offline-workloads\u002Fog.jpg",[54,55,56],"LLM batch APIs: half-price inference for work that can wait","When to use the Anthropic, OpenAI and Gemini batch APIs, how limits, result matching and failures work, and how batch discounts stack with prompt caching.","A wide box holding eight small squares, tiny squares arriving from the left, one long arrow returning.",1792261156302]