> What to check after the model answers: safe Markdown rendering, schema checks before actions, moderation classifiers, streaming rails and groundedness.
>
> Web page: https://balazscsorba.com/blog/llm-output-guardrails-moderation · Language: English · Also available in: [Deutsch](https://balazscsorba.com/de/blog/llm-output-guardrails-moderation.md) · [Magyar](https://balazscsorba.com/hu/blog/llm-output-guardrails-moderation.md)
> Author: Balázs Csorba · Published: 2026-10-17 · Keywords: llm output guardrails, improper output handling llm, owasp llm05, openai moderation api, omni-moderation-latest, llama guard 4, nemo guardrails streaming, markdown image exfiltration, sanitize llm output html, groundedness detection, llm content moderation production, output rails

[Blog](https://balazscsorba.com/blog)/Security & compliance

# LLM output guardrails: moderation, safe rendering and validation before anything acts

What to check after the model answers: safe Markdown rendering, schema checks before actions, moderation classifiers, streaming rails and groundedness.

[Balázs Csorba](https://balazscsorba.com/about)·October 17, 2026·13 min read

-   LLM security
-   Guardrails
-   Content moderation
-   OWASP
-   Output validation

![Three narrowing rectangular frames stacked as a funnel, dots falling through to a tray beside a lock.](https://balazscsorba.com/images/blog/llm-output-guardrails-moderation/cover.webp?v=44213d9401)

## Key takeaways

-   OWASP LLM05:2025 treats model output as untrusted input to the next system: browser, database, shell or mail template.
-   Most real output incidents are rendering and action bugs, not offensive text, so sanitise Markdown and lock down images and links first.
-   Validate structured output against a schema and an allowlist before it triggers any action; never pass it to eval, a shell or string-built SQL.
-   Moderation classifiers such as omni-moderation-latest or Llama Guard 4 give signals, not verdicts; you choose thresholds and actions per category.
-   Streaming changes the trade-off: either the user may see a chunk before it is blocked, or you pay the check latency on every chunk.

On this page

1.  [Model output is untrusted input to the next system](https://balazscsorba.com/#output-is-untrusted-input)
2.  [Four different jobs that all get called guardrails](https://balazscsorba.com/#four-jobs)
3.  [Render Markdown as if an attacker wrote it](https://balazscsorba.com/#render-safely)
4.  [Validate structured output before it does anything](https://balazscsorba.com/#validate-before-acting)
5.  [Moderation classifiers: what the main options check](https://balazscsorba.com/#moderation-classifiers)
6.  [Streaming: see it first or check it first](https://balazscsorba.com/#streaming)
7.  [Groundedness and leakage checks](https://balazscsorba.com/#groundedness-and-leakage)
8.  [Block, redact, rewrite or escalate](https://balazscsorba.com/#actions-and-thresholds)
9.  [Putting it together in one server route](https://balazscsorba.com/#layered-pipeline)
10.  [Testing guardrails like any other feature](https://balazscsorba.com/#testing-guardrails)
11.  [Sources](https://balazscsorba.com/#sources)

Listen to this article

0:000:00

Most guardrail discussions start at the input: prompt injection, jailbreaks, people asking for things they should not get. That half matters, and I wrote about it in [prompt injection defense](https://balazscsorba.com/blog/prompt-injection-lethal-trifecta-patterns). This post is about the other half. Once the model has answered, its text goes somewhere: into a chat bubble rendered as Markdown, into a JSON parser that triggers a refund, into an email template, into a SQL query. Every one of those destinations has its own rules about what is safe, and the model knows none of them.

The OWASP Top 10 for LLM applications names this risk LLM05:2025 Improper Output Handling. Below is how I structure output checks in production features: what to render and how, what to validate before an action, where moderation classifiers fit, what streaming does to all of this, and how to decide between blocking, rewriting and escalating.

## Model output is untrusted input to the next system

OWASP defines improper output handling as insufficient validation, sanitisation and handling of model output before it is passed downstream. The examples in the entry are ordinary web vulnerabilities with a new source: output passed to `exec` or `eval`, JavaScript or Markdown interpreted by the browser as XSS, generated SQL executed without parameterisation, file paths built from output, and content placed in email templates without escaping.

The reason this deserves its own category is that the attacker does not need access to your output. With indirect prompt injection, a document, product review or web page the model reads can steer what it writes. OWASP lists exactly that as a factor that raises the impact, next to model privileges beyond those of the end user and missing context-aware encoding. Its first mitigation is blunt: treat the model as any other user and apply zero trust to what it returns.

## Four different jobs that all get called guardrails

Part of the confusion around output guardrails is that the word covers four jobs with different tools and different owners. I keep them apart in design reviews, because a team that bought a moderation API often believes it has covered all four.

Job

Question it answers

Typical tool

Safe rendering

Can this text run code or leak data when displayed?

Markdown renderer config, sanitiser, CSP

Action validation

Is this tool call or JSON allowed and well-formed?

JSON Schema, allowlists, permission checks

Content safety

Is this harmful, hateful or against policy?

omni-moderation-latest, Llama Guard 4, Azure AI Content Safety

Correctness and leakage

Is it grounded, and does it expose data it should not?

Groundedness checks, PII and secret scanners

The order of that table is also the order in which I would fund the work. Rendering and action bugs turn a single bad answer into an incident; offensive text in a B2B support widget is embarrassing, but it rarely deletes data.

## Render Markdown as if an attacker wrote it

Chat interfaces render Markdown because it looks good. Markdown is also a way to make the browser fetch URLs. The classic exfiltration pattern goes like this: an injected instruction asks the model to put private data into the query string of an image URL, and the browser sends it to the attacker the moment the image renders, without any click. Simon Willison has catalogued it across many products; in the GitLab Duo case reported in May 2025, a hidden instruction in a public project made the assistant fetch a private merge request and exfiltrate a base64 summary through an image tag. GitLab fixed it by allowing links and images only to trusted domains.

In a Vue or Nuxt front end the rules are the ones the Vue security guide already gives for user content: never use untrusted content as a template, avoid `v-html` with unsanitised HTML, and sanitise URLs on the server. For model output I do four things: disable raw HTML in the Markdown renderer, rewrite or drop images and links whose host is not on an allowlist, add `rel="noopener nofollow"` to remaining external links, and send a Content Security Policy whose `img-src` lists only my own hosts. The CSP is the backstop that still works when the renderer has a bug.

```
// server/utils/render-answer.ts — Markdown from the model, rendered on the server
import MarkdownIt from 'markdown-it'

const ALLOWED_HOSTS = new Set(['balazscsorba.com', 'cdn.example-shop.at'])
const md = new MarkdownIt({ html: false, linkify: false })

function allowed(url: string) {
  try {
    const u = new URL(url, 'https://balazscsorba.com')
    return (u.protocol === 'https:' && ALLOWED_HOSTS.has(u.hostname))
  } catch { return false }
}

// images to unknown hosts become plain alt text, so nothing is fetched
md.renderer.rules.image = (tokens, i) => {
  const src = tokens[i].attrGet('src') ?? ''
  return allowed(src) ? `<img src="${md.utils.escapeHtml(src)}" alt="${md.utils.escapeHtml(tokens[i].content)}">`
                      : md.utils.escapeHtml(tokens[i].content)
}
const defaultLink = md.renderer.rules.link_open ?? ((t, i, o, e, s) => s.renderToken(t, i, o))
md.renderer.rules.link_open = (tokens, i, opts, env, self) => {
  if (!allowed(tokens[i].attrGet('href') ?? '')) tokens[i].attrSet('href', '#blocked-link')
  tokens[i].attrSet('rel', 'noopener nofollow')
  return defaultLink(tokens, i, opts, env, self)
}

export const renderAnswer = (text: string) => md.render(text)
```

Render on the server and ship HTML to the client only after these rules ran. If you render in the browser, apply the same rules there and still keep the CSP, because a single component that falls back to `v-html` on raw output undoes the rest.

## Validate structured output before it does anything

When output drives an action, a tool call, a refund, a database query, the checks are deterministic and cheap, so there is no excuse to skip them. Strict JSON Schema mode guarantees shape, not meaning, as I explained in [structured outputs with JSON Schema](https://balazscsorba.com/blog/structured-outputs-json-schema-llm). After parsing, check the values against business rules: is this order number one the current user may see, is this refund below the limit the assistant may grant on its own, is this SKU in the catalogue.

-   Never pass output to `eval`, a shell, or a template engine that executes expressions.
-   Use parameterised queries for every value that came from the model; OWASP lists this explicitly.
-   Resolve file names and IDs against a lookup table instead of building paths from text.
-   Escape output for the context it lands in: HTML, email, CSV, a shell argument.
-   Put a person in front of irreversible actions, as in [human-in-the-loop approval gates](https://balazscsorba.com/blog/human-in-the-loop-ai-agents).

For generated SQL in particular, the read-only role and the query validator matter more than any classifier; the details are in [text-to-SQL guardrails](https://balazscsorba.com/blog/text-to-sql-llm-guardrails).

## Moderation classifiers: what the main options check

For content safety you rarely write your own model. Three families cover most projects I see: the OpenAI moderation endpoint, Meta Llama Guard 4 as an open-weights model you host or rent, and Azure AI Content Safety with several specialised detectors.

Option

What it classifies

Notes from the docs

OpenAI omni-moderation-latest

Text and images; categories such as harassment, hate, illicit, self-harm, sexual, violence

Free to use. Many categories are text only. Can run inline in a generation request.

Meta Llama Guard 4 (12B)

Prompts and responses against hazard categories S1 to S14

Open weights, text and images. Outputs safe or unsafe plus the violated categories. Categories can be customised in the prompt.

Azure AI Content Safety

Harm categories, Prompt Shields, groundedness, protected material

Protected material detection is English only; the code index is current only through April 2023.

Llama Guard 4 uses the MLCommons hazard taxonomy plus one addition: S1 violent crimes, S2 non-violent crimes, S3 sex crimes, S4 child exploitation, S5 defamation, S6 specialized advice, S7 privacy, S8 intellectual property, S9 indiscriminate weapons, S10 hate, S11 self-harm, S12 sexual content, S13 elections and S14 code interpreter abuse. To check a response, Meta says the prompt must contain both the user message and the agent reply, because the question is context for judging the answer. Its first output line is `safe` or `unsafe`, the second lists the violated categories.

OpenAI added inline moderation: pass a top-level `moderation` object with `omni-moderation-latest` in a Responses or Chat Completions request and you get scores for both input and output, without a second call. The docs are explicit about two things I would repeat to every team. A safety-aware refusal can still be flagged because it discusses harmful content, so treat scores as signals for your policy, not as an automatic block. And for tool calls, moderation covers arguments and outputs in the conversation, not tool names, descriptions or schemas.

**A moderation API is not a security boundary**

None of these classifiers checks whether an image URL leaks data, whether a JSON refund is within limits, or whether the answer is true. A clean moderation result on an exfiltration payload is the expected outcome, not a failure of the classifier.

## Streaming: see it first or check it first

Users expect tokens to appear as they are generated, and every output check wants the whole text. You cannot have both, and the tools say so. OpenAI states that inline moderation scores arrive only after the full output is available, not with the deltas. NVIDIA NeMo Guardrails runs output rails on chunks when streaming is enabled: `chunk_size` defaults to 200 tokens, `context_size` carries 50 tokens over between chunks so a sentence that spans a boundary is still judged as a whole, and `stream_first` decides the order.

With `stream_first: True` the client gets each chunk before the rail runs; with `False` the rail runs first and adds its latency to every chunk.

With `stream_first` set to true, which is the default, the user may already have read a chunk that is then blocked, and the stream ends with a JSON error your front end has to handle, ideally by replacing the bubble rather than leaving half a sentence. With false, nothing blocked is ever shown, but each chunk waits for the rail. My rule: stream first for low-risk assistants where the worst case is an awkward sentence; check first when output can contain personal data, prices or commitments the business would have to honour.

## Groundedness and leakage checks

In a RAG assistant the most expensive output failure is a confident answer that the sources do not support. Azure AI Content Safety offers groundedness detection for this: you send the generated text and the grounding sources, choose a domain such as Medical or Generic, and it flags ungrounded content. An optional correction mode returns rewritten text aligned with the sources. I use checks like this as a sampling and alerting tool rather than a hard gate, because a grounding judge is itself a model with errors, and the real fix belongs upstream in retrieval and citations, covered in [grounding and citations](https://balazscsorba.com/blog/llm-hallucination-grounding-citations).

Leakage is easier to check deterministically. Scan output for the patterns you know: IBANs, email addresses and phone numbers of people other than the current user, API keys, internal hostnames, and fragments of your own system prompt. The same detectors you use for [PII redaction](https://balazscsorba.com/blog/pii-redaction-llm-pipelines) on the input side work on the output side; the difference is that on output you usually redact and log instead of rejecting the whole answer.

## Block, redact, rewrite or escalate

A guardrail is a decision, not a score. For each category write down what happens when it fires, and who sees it. This is the table I start with and then tune with real traffic.

Finding

Action

User sees

Unsafe link or image host

Rewrite to text, log

Answer without the link

Clear content-safety violation

Block, replace with fixed text, log scores

Neutral refusal and a contact option

PII or secret of someone else

Redact, alert security

Answer with the value masked

Invalid tool arguments

Reject, return error to the model once

Nothing, unless the retry fails

Ungrounded claim in a RAG answer

Flag for review, sample

Answer with sources, as usual

Two habits keep this honest. Log the classifier version, scores and the action for every decision, because thresholds drift when the provider updates a model behind a `-latest` alias. And show users something useful when you block: a fixed message and a way to reach a person beats a silent empty bubble, which people read as a bug and retry.

## Putting it together in one server route

Cheap deterministic checks run first, classifiers second, and rendering last, so that the expensive and fallible steps see as little as possible. In a Nuxt or any Node backend, the non-streaming version is a short function.

Order by cost: deterministic checks first, the classifier call second, rendering rules last.

```
// server/utils/guard-output.ts
import { renderAnswer } from './render-answer'
import { redactLeaks } from './redact'          // your PII / secret patterns

const BLOCKED = 'I cannot help with that here. Please contact our support team.'

export async function guardOutput(text: string) {
  const { text: clean, findings } = redactLeaks(text)

  const res = await fetch('https://api.openai.com/v1/moderations', {
    method: 'POST',
    headers: { Authorization: `Bearer ${process.env.OPENAI_API_KEY}`, 'Content-Type': 'application/json' },
    body: JSON.stringify({ model: 'omni-moderation-latest', input: clean }),
  })
  if (!res.ok) throw new Error(`moderation failed: ${res.status}`)   // fail closed
  const result = (await res.json()).results[0]

  const decision = result.flagged ? 'block' : findings.length ? 'redact' : 'allow'
  console.info(JSON.stringify({ event: 'output_guard', decision, findings, scores: result.category_scores }))

  return { html: renderAnswer(decision === 'block' ? BLOCKED : clean), decision }
}
```

Two choices in that sketch are deliberate. It fails closed when the moderation call errors, which is right for a public assistant and wrong for an internal tool where availability matters more; decide it explicitly. And it uses `flagged` as the first decision, as OpenAI suggests, while logging every score so you can later move to per-category thresholds. If you prefer a framework, Guardrails AI wraps the same idea in validators you attach to a model call, which I compared in the [Guardrails AI review](https://balazscsorba.com/tools/guardrails-ai).

## Testing guardrails like any other feature

Guardrails fail in two directions, and both cost money. Missed violations are the obvious one. False positives are the quiet one: a support assistant that refuses questions about "killing a process" or "dangerous goods shipping" loses users without anyone noticing. Build a small labelled set with both kinds, including domain vocabulary that trips classifiers, and run it in CI whenever you change a threshold, a prompt or a classifier version.

-   Include exfiltration payloads: image URLs with query strings, links to look-alike domains, HTML tags in Markdown.
-   Include tool calls with out-of-range values and IDs that belong to another customer.
-   Track block rate and false-positive rate per category on production samples, not only in the test set.
-   Red-team the whole chain with a tool such as [Promptfoo](https://balazscsorba.com/tools/promptfoo), since a payload that passes one layer is only interesting if it passes all of them.

Output guardrails are unglamorous engineering: renderer configuration, schemas, a classifier call and a decision table. That is exactly why they work. The model will keep surprising you; the systems after it should not.

## Sources

1.  [OWASP GenAI Security Project: LLM05:2025 Improper Output Handling](https://genai.owasp.org/llmrisk/llm052025-improper-output-handling/)
2.  [OpenAI API Docs: Moderation](https://platform.openai.com/docs/guides/moderation)
3.  [Meta: Llama Guard 4 model card and prompt formats](https://www.llama.com/docs/model-cards-and-prompt-formats/llama-guard-4/)
4.  [NVIDIA NeMo Guardrails: Output rail streaming configuration](https://docs.nvidia.com/nemo/guardrails/configure-guardrails/yaml-schema/streaming/output-rail-streaming.md)
5.  [Microsoft Learn: Prompt Shields in Azure AI Content Safety](https://learn.microsoft.com/en-us/azure/ai-services/content-safety/concepts/jailbreak-detection)
6.  [Microsoft Learn: Groundedness detection in Azure AI Content Safety](https://learn.microsoft.com/en-us/azure/ai-services/content-safety/concepts/groundedness)
7.  [Microsoft Learn: Protected material detection in Azure AI Content Safety](https://learn.microsoft.com/en-us/azure/ai-services/content-safety/concepts/protected-material)
8.  [Guardrails AI documentation](https://www.guardrailsai.com/docs)
9.  [Vue.js docs: Security best practices](https://vuejs.org/guide/best-practices/security.html)
10.  [Simon Willison: Remote prompt injection in GitLab Duo leads to source code theft](https://simonwillison.net/2025/May/23/remote-prompt-injection-in-gitlab-duo/)

## Frequently asked questions

What is improper output handling?

It is the OWASP LLM Top 10 entry LLM05:2025: insufficient validation, sanitisation and handling of model output before it is passed to other components. Typical results are XSS in the browser, SQL injection, path traversal or remote code execution on the backend.

Is the OpenAI moderation endpoint enough as an output guardrail?

It covers content-safety categories such as hate, harassment, self-harm, sexual content, violence and illicit advice, and it is free to use. It does not check whether HTML is safe to render, whether an answer is grounded, or whether a tool call is allowed. Treat it as one layer.

Can I moderate a streamed response?

Yes, but you have to choose. OpenAI returns inline moderation scores only after the full output is available. NeMo Guardrails checks chunks of 200 tokens by default and either streams each chunk before the check or holds it until the check passes.

Should I block or rewrite flagged output?

It depends on the category. Block and replace with a fixed message for clear safety violations, redact for leaked personal data or secrets, and route borderline or business-critical cases to a person. Log every decision with the scores so you can tune thresholds.

How do I stop Markdown image exfiltration?

Do not render images or links to arbitrary domains from model output. Use an allowlist of hosts, strip or rewrite everything else on the server, disable raw HTML in the Markdown renderer and add a strict Content Security Policy.

Written by Balázs Csorba

Senior fullstack & AI engineer in Styria, Austria – 10+ years of Vue, Nuxt, Node.js and PHP, now building tooling for AI agents.

[AI engineering & MCP servers →](https://balazscsorba.com/expertise/ai-engineer)[About me →](https://balazscsorba.com/about)

## More articles

-   [Slopsquatting: when coding agents install packages that do not exist](https://balazscsorba.com/blog/slopsquatting-ai-package-hallucinations)
-   [RAG poisoning: how a few documents can steer your assistant, and how to defend the knowledge base](https://balazscsorba.com/blog/rag-poisoning-knowledge-base-attacks)
-   [Coding agents and secrets: keep keys out of context, logs and commits](https://balazscsorba.com/blog/coding-agent-secrets-hygiene)
-   [AI coding tools and the works council: when usage logs count as monitoring](https://balazscsorba.com/blog/works-council-ai-tools-austria-germany)

## Sounds like what you need?

Tell me about your project or role – I’d love to hear from you.

[Get in touch](mailto:contact@balazscsorba.com) [Connect on LinkedIn](https://www.linkedin.com/in/balazs-csorba)