[{"data":1,"prerenderedAt":607},["ShallowReactive",2],{"tool-promptfoo-en":3},{"slug":4,"published":5,"minutes":6,"category":7,"tags":8,"keywords":13,"about":21,"sources":25,"cover":50,"og":51,"expertise":52,"locales":53,"lang":54,"title":57,"description":58,"coverAlt":59,"url":24,"pricing":60,"kind":61,"metaTitle":62,"takeaways":63,"faq":69,"toc":82,"blocks":106,"others":370},"promptfoo","2026-07-07",10,"llmops",[9,10,11,12],"LLM evaluation","Red teaming","Prompt testing","CI\u002FCD",[4,14,15,16,17,18,19,20],"promptfoo vs langsmith","LLM evaluation framework","promptfoo red teaming","promptfooconfig.yaml example","LLM evals in CI","promptfoo pricing","open source LLM testing",[22],{"name":23,"url":24},"Promptfoo","https:\u002F\u002Fwww.promptfoo.dev",[26,29,32,35,38,41,44,47],{"title":27,"url":28},"Promptfoo documentation: intro","https:\u002F\u002Fwww.promptfoo.dev\u002Fdocs\u002Fintro\u002F",{"title":30,"url":31},"Promptfoo pricing: plan comparison","https:\u002F\u002Fwww.promptfoo.dev\u002Fpricing",{"title":33,"url":34},"Promptfoo documentation: getting started","https:\u002F\u002Fwww.promptfoo.dev\u002Fdocs\u002Fgetting-started\u002F",{"title":36,"url":37},"Promptfoo documentation: command line usage (exit codes)","https:\u002F\u002Fwww.promptfoo.dev\u002Fdocs\u002Fusage\u002Fcommand-line\u002F",{"title":39,"url":40},"Promptfoo documentation: red team plugins","https:\u002F\u002Fwww.promptfoo.dev\u002Fdocs\u002Fred-team\u002Fplugins\u002F",{"title":42,"url":43},"Promptfoo documentation: assertions and metrics","https:\u002F\u002Fwww.promptfoo.dev\u002Fdocs\u002Fconfiguration\u002Fexpected-outputs\u002F",{"title":45,"url":46},"Promptfoo repository on GitHub (MIT licence)","https:\u002F\u002Fgithub.com\u002Fpromptfoo\u002Fpromptfoo",{"title":48,"url":49},"Promptfoo blog: Promptfoo is joining OpenAI (9 March 2026)","https:\u002F\u002Fwww.promptfoo.dev\u002Fblog\u002Fpromptfoo-joining-openai\u002F","\u002Fimages\u002Fblog\u002Fpromptfoo\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fpromptfoo\u002Fog.jpg","ai-engineer",[54,55,56],"en","de","hu","Promptfoo: LLM evals and red teaming from one YAML file","Promptfoo in 2026: MIT-licensed evals and red teaming, now inside OpenAI. What it does well, where the YAML approach breaks, and what the tiers cost.","Diagram: a promptfooconfig.yaml file feeds a runner that calls every provider, assertions grade each output, and the results land in a matrix for the viewer and the CI gate.","MIT · Enterprise paid","Evaluation and red teaming","Promptfoo review: evals and red teaming · Balázs Csorba",[64,65,66,67,68],"Promptfoo is a MIT-licensed CLI that runs quality evals and red teaming from one YAML file, and the repository declared itself part of OpenAI after the March 2026 acquisition announcement.","The Community tier is free and includes 10,000 red team probes a month; dashboards, custom plugins, SSO and the API are quote-only Enterprise rows.","Declarative test cases are the real strength, because a reviewer can read an eval change as a diff without running a Python test suite.","Model-graded assertions such as llm-rubric are nondeterministic and cost a judge call per cell, so the pass rate measures the grader as much as the prompt.","Keep the config and the test data portable: MIT keeps the code forkable, but a model-comparison tool owned by one candidate is a neutrality problem the licence does not solve.",[70,73,76,79],{"q":71,"a":72},"Is promptfoo free to use?","Yes. The Community version is MIT-licensed, runs locally and includes all evaluation features plus up to 10,000 red team probes per month. Enterprise adds custom probe limits, team sharing, continuous monitoring, SSO and API access, and both Enterprise and On-Premise are priced on request.",{"q":74,"a":75},"What is the difference between promptfoo evals and red teaming?","Evals run your prompts against test cases and grade the outputs with assertions such as substring checks, cost thresholds or llm-rubric. Red teaming generates adversarial payloads against a configured target: promptfoo redteam run executes the probes and promptfoo redteam report turns the findings into a vulnerability report.",{"q":77,"a":78},"Which model providers does promptfoo support?","The documentation lists more than 60 providers, including OpenAI, Anthropic, Google, Azure, Bedrock and local models through Ollama. Anything else can be reached through a custom HTTP, Python or JavaScript provider, so the config file does not have to change when the model behind an endpoint does.",{"q":80,"a":81},"What happened to promptfoo after the OpenAI acquisition?","Promptfoo announced on 9 March 2026 that it had agreed to be acquired by OpenAI, stating that the product would remain open source and MIT licensed and that the team would continue to serve customers. The repository now describes promptfoo as part of OpenAI, and the documentation is still being updated.",[83,86,89,92,94,97,100,103],{"id":84,"title":85},"what-it-is","What it is",{"id":87,"title":88},"how-it-works","How it works",{"id":90,"title":91},"getting-started","Getting started",{"id":93,"title":10},"red-teaming",{"id":95,"title":96},"ownership","Who owns it now",{"id":98,"title":99},"where-it-shingles","Where it shingles",{"id":101,"title":102},"verdict","Verdict",{"id":104,"title":105},"sources","Sources",[107,111,114,117,125,151,154,155,162,171,177,178,181,183,192,199,200,218,228,232,252,255,256,259,262,263,266,312,315,321,322,325,338,342,343],{"type":108,"content":109},"paragraph",[110],"Promptfoo is an open-source CLI and library that runs LLM evaluations and adversarial red teaming from a single YAML file. Position up front: it is the most practical way to turn a prompt change or a model swap into a text diff that CI can fail on, and since March 2026 it is also an eval tool that belongs to OpenAI.",{"type":108,"content":112},[113],"It sits between a folder of pytest scripts and a hosted evaluation platform. Compared with LangSmith and Braintrust it gives up dashboards, shared history and a team UI in the free tier, and in exchange it runs on your machine, talks directly to more than 60 providers and needs no account. Compared with DeepEval it gives up Python expressiveness for a config file that a non-programmer can read in review.",{"type":115,"level":116,"id":84,"text":85},"heading",2,{"type":108,"content":118},[119,120,124],"Two products share one repository and one config format. The evaluation side runs prompts against test cases and grades the outputs; the red teaming side attacks a configured target with generated payloads and files the results as a vulnerability report. Both are driven from ",{"tag":121,"children":122},"code",[123],"promptfooconfig.yaml",", both run from the same command line and both can be wired into CI.",{"type":126,"ordered":127,"items":128},"list",false,[129,131,133,135,145,147,149],[130],"MIT-licensed CLI and library: version 0.124.0 on npm, about 25.8k stars and 2.4k forks on GitHub",[132],"Declarative configuration: prompts, providers, tests and assertions in one YAML file",[134],"More than 60 providers, including OpenAI, Anthropic, Google, Azure, Bedrock, Ollama and custom HTTP, Python or JavaScript endpoints",[136,137,140,141,144],"Two commands cover most of the work: ",{"tag":121,"children":138},[139],"promptfoo eval"," for quality, ",{"tag":121,"children":142},[143],"promptfoo redteam run"," for security",[146],"The Community tier is free and includes 10,000 red team probes a month; Enterprise and On-Premise are priced on request",[148],"The vendor reports 350k developers, 130k monthly actives and more than 25% of the Fortune 500",[150],"The site footer carries SOC 2 and ISO 27001 badges, and the documentation ships a GitHub Action for CI",{"type":108,"content":152},[153],"What the free tier deliberately withholds is the hosted part. There is no team dashboard, no searchable scan history, no continuous monitoring and no API in Community; those are Enterprise rows in the feature comparison. The tool itself stays a local process that calls your providers and writes results to your disk.",{"type":115,"level":116,"id":87,"text":88},{"type":108,"content":156},[157,158,161],"An evaluation is a matrix. The config declares prompts, providers and test cases, the runner expands the cross product, sends every cell to every model and scores the response against the assertions attached to that cell. Assertions are typed: substring and JSON schema checks, similarity, cost and latency thresholds, and model-graded checks such as ",{"tag":121,"children":159},[160],"llm-rubric"," that spend an extra model call to judge the output.",{"type":163,"attrs":164,"inner":168,"caption":169},"diagram",{"viewBox":165,"role":166,"aria-labelledby":167},"0 0 730 300","img","t-promptfoo d-promptfoo","\u003Ctitle id=\"t-promptfoo\">How one promptfoo evaluation runs\u003C\u002Ftitle>\u003Cdesc id=\"d-promptfoo\">A four stage pipeline: the config file feeds a runner that calls every provider in parallel with caching, assertions grade each output, and the results go to a web viewer and a CI exit code. Below, a matrix of prompts against providers with pass and fail cells.\u003C\u002Fdesc>\u003Ctext x=\"20\" y=\"26\" class=\"d-title\">One config, one run\u003C\u002Ftext>\u003Ctext x=\"710\" y=\"26\" text-anchor=\"end\" class=\"d-label\">promptfoo eval\u003C\u002Ftext>\u003Crect x=\"20\" y=\"52\" width=\"150\" height=\"56\" rx=\"8\" class=\"d-accent\"\u002F>\u003Ctext x=\"34\" y=\"76\" class=\"d-text\">config\u003C\u002Ftext>\u003Ctext x=\"34\" y=\"96\" class=\"d-small\">prompts, tests\u003C\u002Ftext>\u003Cpath d=\"M170 80 H196\" class=\"d-line\"\u002F>\u003Cpath d=\"M190 74 L198 80 L190 86\" class=\"d-line\"\u002F>\u003Crect x=\"200\" y=\"52\" width=\"150\" height=\"56\" rx=\"8\" class=\"d-box\"\u002F>\u003Ctext x=\"214\" y=\"76\" class=\"d-text\">runner\u003C\u002Ftext>\u003Ctext x=\"214\" y=\"96\" class=\"d-small\">parallel, cached\u003C\u002Ftext>\u003Cpath d=\"M350 80 H376\" class=\"d-line\"\u002F>\u003Cpath d=\"M370 74 L378 80 L370 86\" class=\"d-line\"\u002F>\u003Crect x=\"380\" y=\"52\" width=\"150\" height=\"56\" rx=\"8\" class=\"d-mint\"\u002F>\u003Ctext x=\"394\" y=\"76\" class=\"d-text\">assertions\u003C\u002Ftext>\u003Ctext x=\"394\" y=\"96\" class=\"d-small\">grade each output\u003C\u002Ftext>\u003Cpath d=\"M530 80 H556\" class=\"d-line\"\u002F>\u003Cpath d=\"M550 74 L558 80 L550 86\" class=\"d-line\"\u002F>\u003Crect x=\"560\" y=\"52\" width=\"150\" height=\"56\" rx=\"8\" class=\"d-sky\"\u002F>\u003Ctext x=\"574\" y=\"76\" class=\"d-text\">viewer, CI\u003C\u002Ftext>\u003Ctext x=\"574\" y=\"96\" class=\"d-small\">matrix, exit code\u003C\u002Ftext>\u003Ctext x=\"20\" y=\"152\" class=\"d-label\">RESULTS MATRIX\u003C\u002Ftext>\u003Crect x=\"20\" y=\"164\" width=\"62\" height=\"24\" rx=\"6\" class=\"d-mint\"\u002F>\u003Crect x=\"90\" y=\"164\" width=\"62\" height=\"24\" rx=\"6\" class=\"d-mint\"\u002F>\u003Crect x=\"160\" y=\"164\" width=\"62\" height=\"24\" rx=\"6\" class=\"d-gold\"\u002F>\u003Crect x=\"230\" y=\"164\" width=\"62\" height=\"24\" rx=\"6\" class=\"d-mint\"\u002F>\u003Crect x=\"300\" y=\"164\" width=\"62\" height=\"24\" rx=\"6\" class=\"d-box\"\u002F>\u003Crect x=\"20\" y=\"196\" width=\"62\" height=\"24\" rx=\"6\" class=\"d-mint\"\u002F>\u003Crect x=\"90\" y=\"196\" width=\"62\" height=\"24\" rx=\"6\" class=\"d-gold\"\u002F>\u003Crect x=\"160\" y=\"196\" width=\"62\" height=\"24\" rx=\"6\" class=\"d-mint\"\u002F>\u003Crect x=\"230\" y=\"196\" width=\"62\" height=\"24\" rx=\"6\" class=\"d-mint\"\u002F>\u003Crect x=\"300\" y=\"196\" width=\"62\" height=\"24\" rx=\"6\" class=\"d-box\"\u002F>\u003Crect x=\"20\" y=\"228\" width=\"62\" height=\"24\" rx=\"6\" class=\"d-mint\"\u002F>\u003Crect x=\"90\" y=\"228\" width=\"62\" height=\"24\" rx=\"6\" class=\"d-mint\"\u002F>\u003Crect x=\"160\" y=\"228\" width=\"62\" height=\"24\" rx=\"6\" class=\"d-mint\"\u002F>\u003Crect x=\"230\" y=\"228\" width=\"62\" height=\"24\" rx=\"6\" class=\"d-gold\"\u002F>\u003Crect x=\"300\" y=\"228\" width=\"62\" height=\"24\" rx=\"6\" class=\"d-box\"\u002F>\u003Ctext x=\"394\" y=\"176\" class=\"d-small\">every cell is one prompt x test x provider\u003C\u002Ftext>\u003Ctext x=\"394\" y=\"196\" class=\"d-small\">green passes, gold fails, grey is cached\u003C\u002Ftext>\u003Crect x=\"394\" y=\"212\" width=\"316\" height=\"44\" rx=\"8\" class=\"d-box\"\u002F>\u003Ctext x=\"408\" y=\"234\" class=\"d-text\">exit code gates the pull request\u003C\u002Ftext>\u003Ctext x=\"408\" y=\"250\" class=\"d-small\">100 on a failed test, 1 on any other error\u003C\u002Ftext>",[170],"The promptfoo evaluation matrix: every prompt runs against every provider, assertions grade each cell, and the exit code is what CI reacts to.",{"type":108,"content":172},[173,174,176],"The runner is concurrent and caches completed calls, so a rerun after a small edit mostly re-reads the cache. Results open in a local web viewer as a side-by-side matrix, and the CLI can write JSON, YAML, CSV or HTML. The exit codes are the integration point: ",{"tag":121,"children":175},[139]," returns 100 when at least one test fails or the pass rate falls below PROMPTFOO_PASS_RATE_THRESHOLD, and 1 for any other error.",{"type":115,"level":116,"id":90,"text":91},{"type":108,"content":179},[180],"Install it with npm, brew or pip, or skip the install with npx promptfoo@latest. A minimal config that compares two models on one translated sentence:",{"type":121,"code":182},"# promptfooconfig.yaml: two models, one prompt, three checks\nprompts:\n  - 'Translate this sentence into {{language}}: {{input}}'\n\nproviders:\n  - openai:gpt-6-sol\n  - openai:gpt-6-luna\n\ntests:\n  - vars:\n      language: French\n      input: Hello world\n    assert:\n      - type: contains\n        value: 'Bonjour'\n      - type: llm-rubric\n        value: 'the translation is idiomatic and complete'\n      - type: cost\n        threshold: 0.01",{"type":108,"content":184},[185,187,188,191],{"tag":121,"children":186},[139]," runs every cell of that matrix and prints a pass rate; ",{"tag":121,"children":189},[190],"promptfoo view"," opens the side-by-side comparison in a browser. The -r flag swaps providers from the command line without editing the file, which is how the documentation suggests checking a candidate model against a baseline.",{"type":193,"variant":194,"title":195,"body":196},"callout","tip","Cache before you iterate",[197],[198],"The runner hashes the config, the variables and the provider parameters, so an unchanged cell is not sent to the API again on a rerun. Model-graded assertions are the exception: they are judged at run time, so every one of them costs a judge call per cell.",{"type":115,"level":116,"id":93,"text":10},{"type":108,"content":201},[202,205,206,209,210,213,214,217],{"tag":121,"children":203},[204],"promptfoo redteam init"," writes a red team config, ",{"tag":121,"children":207},[208],"redteam setup"," walks through the target, ",{"tag":121,"children":211},[212],"redteam run"," generates and executes the attacks, and ",{"tag":121,"children":215},[216],"redteam report"," renders the findings. The documentation describes each plugin as a trained model that produces payloads for a specific weakness, rather than a fixed wordlist.",{"type":126,"ordered":127,"items":219},[220,222,224,226],[221],"157 plugins across six categories: brand, compliance and legal, dataset, security and access control, trust and safety, and custom",[223],"Security plugins map to the OWASP Top 10 for LLMs, the OWASP API Security Top 10 and MITRE ATLAS",[225],"Compliance plugins map to NIST AI RMF, ISO\u002FIEC 42001, GDPR Article 5 and EU AI Act Article 5",[227],"Plugins are selected per target in the config, so a scan can be narrowed to the categories one application actually exposes",{"type":115,"level":229,"id":230,"text":231},3,"framework-mappings","Framework mappings",{"type":108,"content":233},[234,235,238,239,238,242,238,245,238,248,251],"For a team that has to show coverage rather than a pile of findings, the framework IDs are the useful part: ",{"tag":121,"children":236},[237],"nist:ai:measure:1.1",", ",{"tag":121,"children":240},[241],"owasp:llm:01",{"tag":121,"children":243},[244],"iso:42001:privacy",{"tag":121,"children":246},[247],"gdpr:art5",{"tag":121,"children":249},[250],"eu:ai-act:art5",". A report grouped by those IDs is a coverage argument a reviewer can follow. This is compliance mechanics rather than a compliance claim: the tool shows which probes ran, not that the product satisfies the regulation.",{"type":108,"content":253},[254],"The free ceiling is 10,000 probes a month, and the plugins that need inference to generate and grade payloads are what consume it. Past that, the pricing page points at Enterprise for custom limits.",{"type":115,"level":116,"id":95,"text":96},{"type":108,"content":257},[258],"On 9 March 2026 Promptfoo announced that it had agreed to be acquired by OpenAI. The post promised that the product would remain open source and MIT licensed and that the team would keep serving customers, and it noted that closing was subject to customary closing conditions. The repository now carries the line that Promptfoo is part of OpenAI, and the documentation was still being updated on 7 October 2026.",{"type":108,"content":260},[261],"The licence keeps the code forkable; it does not keep the roadmap neutral. The engineering worry is narrow and concrete: a tool whose job is to answer which model should we ship is now owned by one of the candidates, and its red teaming, guardrails and model security products sit next to OpenAI's agent platform. Nothing in the MIT licence prevents a fork, and nothing in it obliges the company to prioritise multi-model comparison either.",{"type":115,"level":116,"id":98,"text":99},{"type":108,"content":264},[265],"The first weakness is the config itself. A file with a dozen prompts and forty test cases is readable; one with four hundred cases is not, and anything that needs a loop, a join against production data or a custom parser pushes you into JavaScript assertion callbacks, at which point the language you avoided is back. The second is grading drift: model-graded assertions move between judge versions, so a green suite can turn amber because the grader moved. The third is scope, because a security team that needs an audit trail buys Enterprise regardless of how good the evals are.",{"type":267,"head":268,"rows":277},"table",[269,271,273,275],[270],"Tool",[272],"Interface",[274],"Free tier",[276],"Cost model",[278,286,295,303],[279,280,282,284],[4],[281],"YAML and CLI, runs locally",[283],"Full evals, 10k red team probes a month",[285],"MIT; Enterprise quoted",[287,289,291,293],[288],"LangSmith",[290],"Hosted platform plus SDK",[292],"1 seat and 5k base traces a month",[294],"Plus from 39 USD per seat a month",[296,298,299,301],[297],"Braintrust",[290],[300],"Unlimited users, 1 GB processed data",[302],"Pro at 249 USD a month",[304,306,308,310],[305],"DeepEval",[307],"Python, pytest style",[309],"The whole framework",[311],"Apache 2.0; hosted platform behind it",{"type":108,"content":313},[314],"The commercial shape is a free harness with a paid control plane. Continuous monitoring, a central dashboard, custom plugins, organisation-wide attack profiles, SSO, saved targets, searchable history and the API are Enterprise rows, and on-premise deployment adds a second quote. A solo developer never needs them; a team that has to prove what it tested last quarter does.",{"type":193,"variant":316,"title":317,"body":318},"warn","Grader confidence",[319],[320],"Model-graded assertions report a score, not a measurement. Pin the judge model, keep a small human-labelled sample in the suite and re-run it whenever the judge changes; otherwise the pass rate measures the grader as much as the prompt.",{"type":115,"level":116,"id":101,"text":102},{"type":108,"content":323},[324],"Promptfoo is a good local harness that has grown a serious security half. It is fast, reviewable and honest about what it measures, and the red teaming coverage is now specific enough to argue with. The open question is not the licence, which stays MIT, but who decides which models the tool is tuned to notice.",{"type":126,"ordered":326,"items":327},true,[328,330,332,334,336],[329],"Take it if you want prompt and model changes gated in CI as text diffs, with an exit code that fails the job.",[331],"Take it if you need adversarial testing before release and want it reviewed in the same pull request as your quality evals.",[333],"Skip it if you need a hosted service, because dashboards, shared history, scan search and the API are Enterprise-only.",[335],"Skip it if your evaluation logic needs real code, because a Python framework costs less effort than a YAML file fighting to express a loop.",[337],"Keep your test data portable: the config is MIT and forkable, but a model-comparison tool owned by one of the candidates is a neutrality problem the licence does not solve.",{"type":339,"content":340},"quote",[341],"A suite you can read in a pull request beats a dashboard you have to click through, and promptfoo is built on that premise. Its new owner is the reason to keep the exit code and the test data portable.",{"type":115,"level":116,"id":104,"text":105},{"type":126,"ordered":326,"items":344},[345,349,352,355,358,361,364,367],[346],{"tag":347,"href":28,"children":348},"a",[27],[350],{"tag":347,"href":31,"children":351},[30],[353],{"tag":347,"href":34,"children":354},[33],[356],{"tag":347,"href":37,"children":357},[36],[359],{"tag":347,"href":40,"children":360},[39],[362],{"tag":347,"href":43,"children":363},[42],[365],{"tag":347,"href":46,"children":366},[45],[368],{"tag":347,"href":49,"children":369},[48],[371,420,503,564],{"slug":372,"published":373,"minutes":374,"category":7,"tags":375,"keywords":381,"about":388,"sources":392,"cover":411,"og":412,"expertise":52,"locales":413,"lang":54,"title":414,"description":415,"coverAlt":416,"url":417,"pricing":418,"kind":419},"ollama","2026-09-29",11,[376,377,378,379,380],"Local inference","Open models","llama.cpp","GGUF","Model serving",[372,382,383,384,385,386,387],"ollama vs lm studio","ollama vs vllm","local llm runtime","gguf model server","ollama self hosting","ollama api",[389],{"name":390,"url":391},"Ollama (software)","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FOllama",[393,396,399,402,405,408],{"title":394,"url":395},"Ollama API documentation","https:\u002F\u002Fdocs.ollama.com\u002Fapi",{"title":397,"url":398},"Ollama on GitHub, with the MIT LICENSE file","https:\u002F\u002Fgithub.com\u002Follama\u002Follama",{"title":400,"url":401},"Ollama terms of service, last updated May 2026","https:\u002F\u002Follama.com\u002Fterms",{"title":403,"url":404},"Ollama pricing, cloud plans and per-token model rates","https:\u002F\u002Follama.com\u002Fpricing",{"title":406,"url":407},"Hardware support: Nvidia, AMD, Metal and Vulkan","https:\u002F\u002Fdocs.ollama.com\u002Fgpu",{"title":409,"url":410},"OpenAI compatibility, including what is not supported","https:\u002F\u002Fdocs.ollama.com\u002Fapi\u002Fopenai-compatibility","\u002Fimages\u002Fblog\u002Follama\u002Fcover.webp","\u002Fimages\u002Fblog\u002Follama\u002Fog.jpg",[54,55,56],"Ollama review: the friendly way to run open models","Ollama serves open models over one HTTP API on your own hardware. What it does well, where throughput falls short, and what the MIT licence does not cover.","Abstract cover art for the Ollama review","https:\u002F\u002Follama.com","MIT · free for personal use","Local inference runtime",{"slug":421,"published":422,"minutes":374,"category":7,"tags":423,"keywords":429,"about":437,"sources":444,"cover":496,"og":497,"expertise":52,"locales":498,"lang":54,"title":499,"description":500,"coverAlt":501,"url":440,"pricing":502,"kind":424},"portkey","2026-09-28",[424,425,426,427,428],"LLM gateway","Guardrails","Routing","Observability","Cost control",[430,431,432,433,434,435,436],"portkey ai gateway","portkey vs litellm","llm gateway comparison","llm gateway latency overhead","llm guardrails gateway","self-hosted llm gateway","portkey pricing",[438,441],{"name":439,"url":440},"Portkey","https:\u002F\u002Fportkey.ai",{"name":442,"url":443},"API gateway","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FAPI_gateway",[445,448,451,454,457,460,463,466,469,472,475,478,481,484,487,490,493],{"title":446,"url":447},"Portkey docs: AI Gateway","https:\u002F\u002Fportkey.ai\u002Fdocs\u002Fproduct\u002Fai-gateway",{"title":449,"url":450},"Portkey docs: Getting started with the AI Gateway","https:\u002F\u002Fdocs.portkey.ai\u002Fdocs\u002Fguides\u002Fgetting-started\u002Fgetting-started-with-ai-gateway",{"title":452,"url":453},"Portkey docs: Gateway config object","https:\u002F\u002Fportkey.ai\u002Fdocs\u002Fapi-reference\u002Fconfig-object",{"title":455,"url":456},"Portkey docs: Guardrails","https:\u002F\u002Fportkey.ai\u002Fdocs\u002Fproduct\u002Fguardrails",{"title":458,"url":459},"Portkey docs: Guardrail endpoints and capabilities","https:\u002F\u002Fportkey.ai\u002Fdocs\u002Fproduct\u002Fguardrails\u002Fcapabilities",{"title":461,"url":462},"Portkey docs: Cache, simple and semantic","https:\u002F\u002Fportkey.ai\u002Fdocs\u002Fproduct\u002Fai-gateway\u002Fcache-simple-and-semantic",{"title":464,"url":465},"Portkey docs: Load balancing","https:\u002F\u002Fportkey.ai\u002Fdocs\u002Fproduct\u002Fai-gateway\u002Fload-balancing",{"title":467,"url":468},"Portkey docs: Enterprise hybrid deployment architecture","https:\u002F\u002Fportkey.ai\u002Fdocs\u002Fself-hosting\u002Fhybrid-deployments\u002Farchitecture",{"title":470,"url":471},"Portkey pricing","https:\u002F\u002Fportkey.ai\u002Fpricing",{"title":473,"url":474},"Portkey gateway on GitHub, MIT licensed","https:\u002F\u002Fgithub.com\u002FPortkey-AI\u002Fgateway",{"title":476,"url":477},"Portkey's own benchmark: gateway versus direct Bedrock","https:\u002F\u002Fgithub.com\u002FPortkey-AI\u002Fbenchmark-test",{"title":479,"url":480},"Portkey status page","https:\u002F\u002Fstatus.portkey.ai\u002F",{"title":482,"url":483},"Palo Alto Networks completes acquisition of Portkey, May 2026","https:\u002F\u002Fwww.paloaltonetworks.com\u002Fcompany\u002Fpress\u002F2026\u002Fpalo-alto-networks-completes-acquisition-of-portkey-to-secure-ai-agents",{"title":485,"url":486},"Palo Alto Networks: Prisma AIRS AI Gateway","https:\u002F\u002Fwww.paloaltonetworks.com\u002Fai-security\u002Fai-gateway",{"title":488,"url":489},"Cloudflare AI Gateway pricing","https:\u002F\u002Fdevelopers.cloudflare.com\u002Fai-gateway\u002Freference\u002Fpricing\u002F",{"title":491,"url":492},"LiteLLM pricing","https:\u002F\u002Fwww.litellm.ai\u002Fpricing",{"title":494,"url":495},"OpenRouter pricing","https:\u002F\u002Fopenrouter.ai\u002Fpricing","\u002Fimages\u002Fblog\u002Fportkey\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fportkey\u002Fog.jpg",[54,55,56],"Portkey: a production LLM gateway, reviewed for routing, guardrails and cost","Portkey puts retries, fallbacks, caching, guardrails and cost tracking behind one OpenAI-compatible endpoint. What the config object does well, what the gateway costs in latency, and when to self-host.","A request path from an application through the Portkey gateway to three model providers, with the guardrail verdict and the log written below the proxy.","Free · from $49 per month",{"slug":504,"published":505,"minutes":6,"category":7,"tags":506,"keywords":512,"about":520,"sources":532,"cover":557,"og":558,"expertise":52,"locales":559,"lang":54,"title":560,"description":561,"coverAlt":562,"url":523,"pricing":563,"kind":507},"langfuse","2026-08-13",[507,508,509,510,511],"LLM observability","Tracing","OpenTelemetry","Self-hosting","Evaluation",[504,513,514,515,516,517,518,519],"langfuse vs langsmith","llm tracing tool","self-hosted llm observability","langfuse pricing","opentelemetry llm traces","llm cost tracking","prompt versioning",[521,524,526,529],{"name":522,"url":523},"Langfuse","https:\u002F\u002Flangfuse.com",{"name":509,"url":525},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FOpenTelemetry",{"name":527,"url":528},"ClickHouse","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FClickHouse",{"name":530,"url":531},"Observability (software)","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FObservability_(software)",[533,536,539,542,545,548,551,554],{"title":534,"url":535},"Langfuse documentation: observability and application tracing","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fobservability\u002Foverview",{"title":537,"url":538},"Langfuse documentation: get started with tracing","https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fobservability\u002Fget-started",{"title":540,"url":541},"Langfuse pricing: cloud plans, billable units and worked examples","https:\u002F\u002Flangfuse.com\u002Fpricing",{"title":543,"url":544},"Langfuse pricing: self-hosted plans and the feature comparison","https:\u002F\u002Flangfuse.com\u002Fpricing-self-host",{"title":546,"url":547},"Self-host Langfuse: deployment options, containers and storage services","https:\u002F\u002Flangfuse.com\u002Fself-hosting",{"title":549,"url":550},"Langfuse changelog: v4 is live (17 August 2026)","https:\u002F\u002Flangfuse.com\u002Fchangelog\u002F2026-08-17-langfuse-v4",{"title":552,"url":553},"Langfuse blog: Langfuse joins ClickHouse (16 January 2026)","https:\u002F\u002Flangfuse.com\u002Fblog\u002Fjoining-clickhouse",{"title":555,"url":556},"GitHub: langfuse\u002Flangfuse, the platform repository","https:\u002F\u002Fgithub.com\u002Flangfuse\u002Flangfuse","\u002Fimages\u002Fblog\u002Flangfuse\u002Fcover.webp","\u002Fimages\u002Fblog\u002Flangfuse\u002Fog.jpg",[54,55,56],"Langfuse review: tracing, prompts and evals you can host yourself","Langfuse puts LLM traces, prompt versions and experiments on one MIT-licensed platform. What self-hosting really costs, how the unit pricing adds up, and where it loses.","A pipeline from a batched application event through the Langfuse web container and object storage into ClickHouse, with Redis and PostgreSQL alongside.","MIT · paid from $59 per month",{"slug":565,"published":566,"minutes":6,"category":7,"tags":567,"keywords":572,"about":579,"sources":583,"cover":599,"og":600,"expertise":52,"locales":601,"lang":54,"title":602,"description":603,"coverAlt":604,"url":605,"pricing":606,"kind":424},"openrouter","2026-07-23",[424,568,569,570,571],"Model routing","Fallbacks","OpenAI-compatible","Pay per token",[565,573,432,574,575,576,577,578],"openrouter vs litellm","openrouter pricing","openai compatible api gateway","llm fallback routing","multi model api gateway","byok llm routing",[580],{"name":581,"url":582},"OpenRouter","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FOpenRouter",[584,587,588,591,594,597],{"title":585,"url":586},"OpenRouter documentation: quickstart","https:\u002F\u002Fopenrouter.ai\u002Fdocs\u002Fquickstart",{"title":494,"url":495},{"title":589,"url":590},"OpenRouter documentation: model fallbacks","https:\u002F\u002Fopenrouter.ai\u002Fdocs\u002Fguides\u002Frouting\u002Fmodel-fallbacks",{"title":592,"url":593},"OpenRouter documentation: provider routing","https:\u002F\u002Fopenrouter.ai\u002Fdocs\u002Fguides\u002Frouting\u002Fprovider-selection",{"title":595,"url":596},"OpenRouter documentation index","https:\u002F\u002Fopenrouter.ai\u002Fdocs\u002Fllms.txt",{"title":598,"url":582},"Wikipedia: OpenRouter","\u002Fimages\u002Fblog\u002Fopenrouter\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fopenrouter\u002Fog.jpg",[54,55,56],"OpenRouter: one API key in front of every model you might call","OpenRouter puts 500+ models from 80+ providers behind one OpenAI-compatible endpoint, with fallbacks and pass-through pricing. What it costs, where it breaks.","Request path through OpenRouter: client, router, candidate providers, fallback list and the model that finally answers.","https:\u002F\u002Fopenrouter.ai","Pay per token, no subscription",1791383548914]