LLM output guardrails: moderation, safe rendering and validation before anything acts

Most guardrail discussions start at the input: prompt injection, jailbreaks, people asking for things they should not get. That half matters, and I wrote about it in prompt injection defense. This post is about the other half. Once the model has answered, its text goes somewhere: into a chat bubble rendered as Markdown, into a JSON parser that triggers a refund, into an email template, into a SQL query. Every one of those destinations has its own rules about what is safe, and the model knows none of them.

The OWASP Top 10 for LLM applications names this risk LLM05:2025 Improper Output Handling. Below is how I structure output checks in production features: what to render and how, what to validate before an action, where moderation classifiers fit, what streaming does to all of this, and how to decide between blocking, rewriting and escalating.

Model output is untrusted input to the next system

OWASP defines improper output handling as insufficient validation, sanitisation and handling of model output before it is passed downstream. The examples in the entry are ordinary web vulnerabilities with a new source: output passed to exec or eval, JavaScript or Markdown interpreted by the browser as XSS, generated SQL executed without parameterisation, file paths built from output, and content placed in email templates without escaping.

The reason this deserves its own category is that the attacker does not need access to your output. With indirect prompt injection, a document, product review or web page the model reads can steer what it writes. OWASP lists exactly that as a factor that raises the impact, next to model privileges beyond those of the end user and missing context-aware encoding. Its first mitigation is blunt: treat the model as any other user and apply zero trust to what it returns.

Four different jobs that all get called guardrails

Part of the confusion around output guardrails is that the word covers four jobs with different tools and different owners. I keep them apart in design reviews, because a team that bought a moderation API often believes it has covered all four.

The order of that table is also the order in which I would fund the work. Rendering and action bugs turn a single bad answer into an incident; offensive text in a B2B support widget is embarrassing, but it rarely deletes data.

Render Markdown as if an attacker wrote it

Chat interfaces render Markdown because it looks good. Markdown is also a way to make the browser fetch URLs. The classic exfiltration pattern goes like this: an injected instruction asks the model to put private data into the query string of an image URL, and the browser sends it to the attacker the moment the image renders, without any click. Simon Willison has catalogued it across many products; in the GitLab Duo case reported in May 2025, a hidden instruction in a public project made the assistant fetch a private merge request and exfiltrate a base64 summary through an image tag. GitLab fixed it by allowing links and images only to trusted domains.

In a Vue or Nuxt front end the rules are the ones the Vue security guide already gives for user content: never use untrusted content as a template, avoid v-html with unsanitised HTML, and sanitise URLs on the server. For model output I do four things: disable raw HTML in the Markdown renderer, rewrite or drop images and links whose host is not on an allowlist, add rel="noopener nofollow" to remaining external links, and send a Content Security Policy whose img-src lists only my own hosts. The CSP is the backstop that still works when the renderer has a bug.

// server/utils/render-answer.ts — Markdown from the model, rendered on the server
import MarkdownIt from 'markdown-it'

const ALLOWED_HOSTS = new Set(['balazscsorba.com', 'cdn.example-shop.at'])
const md = new MarkdownIt({ html: false, linkify: false })

function allowed(url: string) {
  try {
    const u = new URL(url, 'https://balazscsorba.com')
    return (u.protocol === 'https:' && ALLOWED_HOSTS.has(u.hostname))
  } catch { return false }
}

// images to unknown hosts become plain alt text, so nothing is fetched
md.renderer.rules.image = (tokens, i) => {
  const src = tokens[i].attrGet('src') ?? ''
  return allowed(src) ? `<img src="${md.utils.escapeHtml(src)}" alt="${md.utils.escapeHtml(tokens[i].content)}">`
                      : md.utils.escapeHtml(tokens[i].content)
}
const defaultLink = md.renderer.rules.link_open ?? ((t, i, o, e, s) => s.renderToken(t, i, o))
md.renderer.rules.link_open = (tokens, i, opts, env, self) => {
  if (!allowed(tokens[i].attrGet('href') ?? '')) tokens[i].attrSet('href', '#blocked-link')
  tokens[i].attrSet('rel', 'noopener nofollow')
  return defaultLink(tokens, i, opts, env, self)
}

export const renderAnswer = (text: string) => md.render(text)

Render on the server and ship HTML to the client only after these rules ran. If you render in the browser, apply the same rules there and still keep the CSP, because a single component that falls back to v-html on raw output undoes the rest.

Validate structured output before it does anything

When output drives an action, a tool call, a refund, a database query, the checks are deterministic and cheap, so there is no excuse to skip them. Strict JSON Schema mode guarantees shape, not meaning, as I explained in structured outputs with JSON Schema. After parsing, check the values against business rules: is this order number one the current user may see, is this refund below the limit the assistant may grant on its own, is this SKU in the catalogue.

• Never pass output to eval, a shell, or a template engine that executes expressions.
• Use parameterised queries for every value that came from the model; OWASP lists this explicitly.
• Resolve file names and IDs against a lookup table instead of building paths from text.
• Escape output for the context it lands in: HTML, email, CSV, a shell argument.
• Put a person in front of irreversible actions, as in human-in-the-loop approval gates.

For generated SQL in particular, the read-only role and the query validator matter more than any classifier; the details are in text-to-SQL guardrails.

Moderation classifiers: what the main options check

For content safety you rarely write your own model. Three families cover most projects I see: the OpenAI moderation endpoint, Meta Llama Guard 4 as an open-weights model you host or rent, and Azure AI Content Safety with several specialised detectors.

Llama Guard 4 uses the MLCommons hazard taxonomy plus one addition: S1 violent crimes, S2 non-violent crimes, S3 sex crimes, S4 child exploitation, S5 defamation, S6 specialized advice, S7 privacy, S8 intellectual property, S9 indiscriminate weapons, S10 hate, S11 self-harm, S12 sexual content, S13 elections and S14 code interpreter abuse. To check a response, Meta says the prompt must contain both the user message and the agent reply, because the question is context for judging the answer. Its first output line is safe or unsafe, the second lists the violated categories.

OpenAI added inline moderation: pass a top-level moderation object with omni-moderation-latest in a Responses or Chat Completions request and you get scores for both input and output, without a second call. The docs are explicit about two things I would repeat to every team. A safety-aware refusal can still be flagged because it discusses harmful content, so treat scores as signals for your policy, not as an automatic block. And for tool calls, moderation covers arguments and outputs in the conversation, not tool names, descriptions or schemas.

Streaming: see it first or check it first

Users expect tokens to appear as they are generated, and every output check wants the whole text. You cannot have both, and the tools say so. OpenAI states that inline moderation scores arrive only after the full output is available, not with the deltas. NVIDIA NeMo Guardrails runs output rails on chunks when streaming is enabled: chunk_size defaults to 200 tokens, context_size carries 50 tokens over between chunks so a sentence that spans a boundary is still judged as a whole, and stream_first decides the order.

With stream_first set to true, which is the default, the user may already have read a chunk that is then blocked, and the stream ends with a JSON error your front end has to handle, ideally by replacing the bubble rather than leaving half a sentence. With false, nothing blocked is ever shown, but each chunk waits for the rail. My rule: stream first for low-risk assistants where the worst case is an awkward sentence; check first when output can contain personal data, prices or commitments the business would have to honour.

Groundedness and leakage checks

In a RAG assistant the most expensive output failure is a confident answer that the sources do not support. Azure AI Content Safety offers groundedness detection for this: you send the generated text and the grounding sources, choose a domain such as Medical or Generic, and it flags ungrounded content. An optional correction mode returns rewritten text aligned with the sources. I use checks like this as a sampling and alerting tool rather than a hard gate, because a grounding judge is itself a model with errors, and the real fix belongs upstream in retrieval and citations, covered in grounding and citations.

Leakage is easier to check deterministically. Scan output for the patterns you know: IBANs, email addresses and phone numbers of people other than the current user, API keys, internal hostnames, and fragments of your own system prompt. The same detectors you use for PII redaction on the input side work on the output side; the difference is that on output you usually redact and log instead of rejecting the whole answer.

Block, redact, rewrite or escalate

A guardrail is a decision, not a score. For each category write down what happens when it fires, and who sees it. This is the table I start with and then tune with real traffic.

Two habits keep this honest. Log the classifier version, scores and the action for every decision, because thresholds drift when the provider updates a model behind a -latest alias. And show users something useful when you block: a fixed message and a way to reach a person beats a silent empty bubble, which people read as a bug and retry.

Putting it together in one server route

Cheap deterministic checks run first, classifiers second, and rendering last, so that the expensive and fallible steps see as little as possible. In a Nuxt or any Node backend, the non-streaming version is a short function.

// server/utils/guard-output.ts
import { renderAnswer } from './render-answer'
import { redactLeaks } from './redact'          // your PII / secret patterns

const BLOCKED = 'I cannot help with that here. Please contact our support team.'

export async function guardOutput(text: string) {
  const { text: clean, findings } = redactLeaks(text)

  const res = await fetch('https://api.openai.com/v1/moderations', {
    method: 'POST',
    headers: { Authorization: `Bearer ${process.env.OPENAI_API_KEY}`, 'Content-Type': 'application/json' },
    body: JSON.stringify({ model: 'omni-moderation-latest', input: clean }),
  })
  if (!res.ok) throw new Error(`moderation failed: ${res.status}`)   // fail closed
  const result = (await res.json()).results[0]

  const decision = result.flagged ? 'block' : findings.length ? 'redact' : 'allow'
  console.info(JSON.stringify({ event: 'output_guard', decision, findings, scores: result.category_scores }))

  return { html: renderAnswer(decision === 'block' ? BLOCKED : clean), decision }
}

Two choices in that sketch are deliberate. It fails closed when the moderation call errors, which is right for a public assistant and wrong for an internal tool where availability matters more; decide it explicitly. And it uses flagged as the first decision, as OpenAI suggests, while logging every score so you can later move to per-category thresholds. If you prefer a framework, Guardrails AI wraps the same idea in validators you attach to a model call, which I compared in the Guardrails AI review.

Testing guardrails like any other feature

Guardrails fail in two directions, and both cost money. Missed violations are the obvious one. False positives are the quiet one: a support assistant that refuses questions about "killing a process" or "dangerous goods shipping" loses users without anyone noticing. Build a small labelled set with both kinds, including domain vocabulary that trips classifiers, and run it in CI whenever you change a threshold, a prompt or a classifier version.

• Include exfiltration payloads: image URLs with query strings, links to look-alike domains, HTML tags in Markdown.
• Include tool calls with out-of-range values and IDs that belong to another customer.
• Track block rate and false-positive rate per category on production samples, not only in the test set.
• Red-team the whole chain with a tool such as Promptfoo, since a payload that passes one layer is only interesting if it passes all of them.

Output guardrails are unglamorous engineering: renderer configuration, schemas, a classifier call and a decision table. That is exactly why they work. The model will keep surprising you; the systems after it should not.

Sources

1. OWASP GenAI Security Project: LLM05:2025 Improper Output Handling
2. OpenAI API Docs: Moderation
3. Meta: Llama Guard 4 model card and prompt formats
4. NVIDIA NeMo Guardrails: Output rail streaming configuration
5. Microsoft Learn: Prompt Shields in Azure AI Content Safety
6. Microsoft Learn: Groundedness detection in Azure AI Content Safety
7. Microsoft Learn: Protected material detection in Azure AI Content Safety
8. Guardrails AI documentation
9. Vue.js docs: Security best practices
10. Simon Willison: Remote prompt injection in GitLab Duo leads to source code theft