Generating synthetic test data for RAG evaluations

Every RAG team hits the same wall: the code ships quickly, and the evaluation dataset does not. Labeling question, answer and gold-context triples by hand takes weeks, and by the time the set exists the knowledge base has moved on. Red Hat's February 2026 write-up describes the usual state of affairs: teams tune embeddings, retrievers, chunking and prompts while still relying on manual spot checks, small hand-labeled datasets or generic LLM-as-a-judge scores, and the system appears to work until real traffic arrives.

Synthetic test data is what the industry converged on instead: generate the questions, answers and gold contexts from your own chunks, filter what the generator got wrong, and keep a small human sample to grade the generator itself. This post walks the full path: the pipeline, the question shapes worth producing, hard negatives, answerability labels, German and Hungarian coverage, contamination, spot checks, and the metrics that turn the result into a CI gate instead of a dashboard.

Why hand-labeled sets run out

RAG evaluation is harder than evaluating a model on its own for a boring reason: two components fail at once. Without known ground-truth context you cannot tell whether the retriever surfaced the wrong documents or the generator hallucinated despite correct context, and debugging turns into guesswork. Synthetic triplets fix exactly this: because the question was built from a specific chunk, you know which chunk should win, so retrieval and generation can be scored separately instead of arguing about one blended score.

Red Hat lists the other failure modes of static, hand-made sets: they are expensive and slow to produce, they rarely carry ground-truth context, they go stale as soon as the documents change, and generic questions miss the domain-specific failures that actually matter. There is a caveat worth keeping in mind. Rahmani and colleagues showed at CIKM 2025 that evaluation results computed on LLM-generated test collections carry bias for absolute system performance, while relative comparisons between systems hold up much better. Read synthetic scores as a trend line, not as a grade.

The generation pipeline

Every serious generator follows the same shape. Red Hat's open source SDG Hub ships a RAG evaluation flow with six steps: topic extraction to anchor the evaluation, question generation guided by a document outline, question evolution into realistic user-style queries, answer generation grounded strictly in the source, groundedness filtering that drops weak pairs, and context extraction that isolates the minimal chunk needed to answer. The input contract is deliberate: one column for the document, which must match your production chunking strategy, and one short intent-level outline label, because a good outline is what stops the model from writing trivial extractive questions.

Ragas builds the same output from a different direction. It turns your documents into a knowledge graph using a document splitter, extractors such as entity and keyphrase extractors, and relationship builders, then picks scenarios, meaning the nodes, query length, query style and persona, before a synthesizer writes the question and its reference answer. Its default query distribution is 0.5 single-hop specific, 0.25 multi-hop abstract and 0.25 multi-hop specific. DeepEval's Synthesizer takes the plain route: generate goldens from documents or contexts, evolve them into harder shapes, and store them as a dataset you can run pytest against.

from ragas.testset import TestsetGenerator
from ragas.testset.synthesizers import default_query_distribution

generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings)
dist = default_query_distribution(generator_llm)
# [(SingleHopSpecific, 0.5), (MultiHopAbstract, 0.25), (MultiHopSpecific, 0.25)]
testset = generator.generate(testset_size=100, query_distribution=dist)

df = testset.to_pandas()
# columns you actually need downstream: query, reference_contexts, reference
df.to_json("evals/rag-set-v1.jsonl", orient="records", lines=True)

Question shapes worth generating

Do not generate one kind of question. A set made only of what-is-X items will happily pass while the failures that reach production, the comparison across two documents, the paraphrase with no shared keywords, and the question your corpus simply cannot answer, never appear at all. Ragas splits along two axes, single-hop versus multi-hop and specific versus abstract, and its scenarios add query length, style and persona so the wording does not come out in a single uniform register.

Difficulty is the second axis, and it is measurable. NVIDIA's synthetic-data pipeline uses an embedding model as a judge for question easiness: it filters generated questions by cosine similarity to the context documents, and lower thresholds produce a harder set. On the FiQA benchmark the NVIDIA team calibrated that filter against the human-annotated test queries and reported an average deviation in recall at 5 of 4.57 percent with a 70th-percentile threshold and 5.65 percent with a 75th-percentile threshold, with a 0.91 correlation across the embedding models they compared. That is the standard worth aiming for: your synthetic set should rank two retrievers in the same order as a human-labeled one.

Hard negatives: tests that can actually fail

A question plus its gold chunk only proves that the right chunk can be found. Hard negatives prove that the wrong chunk is rejected, and they are the main reason a synthetic set can fail at all. NVIDIA defines hard negatives as documents whose embeddings sit close to the question but whose text does not contain the answer: the neighbouring price list, the previous version of a policy, the same article in the other language. Random negatives are useless for this, because the distance from a question to an irrelevant document is already large in a modern embedding space.

For building training pairs NVIDIA documents three selection strategies, and all three apply to evaluation data: top-K selection by cosine similarity to the question, threshold-based selection that only accepts negatives inside a minimum and maximum similarity band, and positive-aware mining, which caps the negative similarity as a percentage of the positive score. Their ablation in the NV-Retriever paper sets that cap at 95 percent of the positive score to remove false negatives. For evals the same three options mean: pick the decoys a retriever is most likely to confuse with the gold chunk, then store their ids next to the label so a failure points at a specific confusion instead of a vague score drop.

import numpy as np

def hard_negatives(question, chunks, gold_idx, k=4, cap=0.95):
    """Decoys the retriever ranks near the gold chunk, capped below the gold score."""
    q = embed(question)
    sims = np.array([cosine(q, c.embedding) for c in chunks])
    gold = sims[gold_idx]
    order = np.argsort(-sims)
    picked = [i for i in order if i != gold_idx and sims[i] <= cap * gold]
    return picked[:k]  # store chunk ids as hard_negatives on the record

Answerability labels

Unanswerable items are what separate a search test from a refusal test. SQuAD 2.0 made the pattern standard by adding over 50,000 questions written adversarially by crowdworkers to look similar to answerable ones: a strong neural system that scored 86 percent F1 on SQuAD 1.1 dropped to 66 percent on SQuAD 2.0, because it also had to decide when no answer is supported by the paragraph. A RAG eval set needs the same mixture, with a documented share of questions where the correct behaviour is to say the corpus does not cover the topic.

NVIDIA's answerability filter is a good template for the label itself. Human annotators scored a generated query A when the chunk completely answered it, B when key information was missing, C when the query was relevant but the context did not answer it, and D when the query was not relevant to the context, with no score at all when the query was not standalone. Against that rubric, a Llama-3.1-70b-Instruct judge reached 94 percent precision and 90 percent recall. Four grades beat a boolean here, because partially answerable is a real production case, and it is the one that produces confident, half-right answers.

German and Hungarian coverage

If the product answers in German and Hungarian, generate in German and Hungarian. Translating an English question set gives you English phrasing with German words, which is not what people type, and it hides the retrieval problems that are specific to each language: German compound nouns that split differently from the query, Hungarian agglutination that hangs several suffixes on one stem, and the cross-lingual case where a Hungarian question should hit a German document. Label every item with its query language and document language, and score the cross-lingual slice on its own.

Public multilingual benchmarks are a sanity check, not a substitute. MMTEB, the community expansion of MTEB accepted at ICLR 2025, covers over 500 quality-controlled evaluation tasks across more than 250 languages, and it also shows how much task selection matters: retrieval tasks are optimized by sampling hard negatives to build smaller splits that preserve the model ranking. National extensions of MTEB exist for German and several other languages, and they help when you pick an embedding model; for Hungarian you will mostly rely on your own generated set, which is one more reason to keep it honest. The retrieval side of this is covered in multilingual RAG for German and Hungarian.

Keeping the set uncontaminated

Contamination bites twice. The first bite is the public-benchmark kind: if evaluation items appear in a model's training data, the score is fiction. The classic detector is n-gram overlap, the method the GPT-3 paper used to flag evaluation sets with more than 13-token overlap against training data, and the Infini-gram mini study at EMNLP 2025 ran exact n-gram search over 83 terabytes of Internet text and found core benchmarks heavily contaminated, up to 74.2 percent for GSM8K. A 2025 survey groups the defences into data updating, data rewriting and prevention, and the detection methods into white-box, gray-box and black-box.

The second bite is private: your generated set leaking into the system you are measuring. Three rules cover it. Do not tune a prompt or a chunking rule on the eval set and then report a number on that same set, so hold out whole documents rather than chunks for tuning work and keep the eval records out of the index used for prompt experiments. Check overlap between each generated question and its gold chunk, because a question copied from the chunk wording makes an extractive retriever look better than it is. And regenerate when the corpus changes materially, storing a corpus hash on every item so an old score is traceable.

Human spot checks that scale

Spot checks are how you grade the generator, and the cheap version works. Evidently's hands-on guide to aligning a judge with human labels suggests starting with 30 to 50 labeled examples, keeping labels binary or few-class instead of one-to-ten scales, and measuring the judge with accuracy, precision, recall and Cohen's kappa. In their worked example the first judge prompt agreed with human experts on only 67 percent of labels, and it took prompt and model iteration before the judge was usable. Budget for that iteration instead of assuming the first judge prompt is final.

Sample stratified rather than randomly: cover every language, every label type and the unanswerable slice, and re-check after every change to the generator model or its prompt. There is also a specific bias to guard against. Panickssery, Bowman and Feng showed that LLM evaluators recognise and favour their own generations, and that fine-tuning experiments produce a linear correlation between self-recognition capability and the strength of the self-preference bias. Practically: never grade questions and answers with the same model family that wrote them, and refresh judge agreement as a measured number instead of a property you assume.

What the numbers must measure

With gold contexts in hand, the metric list is short and every entry names a layer, which is the whole point of building the set this way. Retrieval gets recall at k, MRR and nDCG, plus context precision and recall when you know the answer spans and not only the chunk ids. Generation gets faithfulness, meaning the answer is supported by the retrieved context, answer relevance, and abstention accuracy computed over the unanswerable items. A number you cannot attribute to one of the two layers is a dashboard, not a diagnostic; the wider taxonomy is in evaluating RAG.

Two habits keep the numbers honest. Report the relative comparison, configuration A against configuration B on the same set, as the primary result, because the CIKM 2025 analysis found absolute performance on synthetic collections to be the fragile number. And keep a small human-labeled slice permanently: it is the calibration weight that tells you whether the synthetic set has drifted away from what users actually ask, and it doubles as the regression set for the judge.

Making it a repeatable job

Synthetic data only pays off when it is boring: the same pipeline, the same prompts and the same filters, running whenever the corpus changes enough to matter. Give every item a corpus hash, a generator model id and a prompt version, so a score you cannot reproduce is traceable to the change that caused it. The loop below is the whole practice: generate, filter, spot-check a stratified sample, run the evaluation, compare against the previous baseline, and regenerate only when the data has moved.

{
  "id": "hu-0042",
  "corpus_hash": "8f3c1a9",
  "generator": "<model id>",
  "prompt_version": "qgen-v4",
  "query_lang": "hu",
  "doc_lang": "de",
  "question": "<generated question>",
  "answer": "<answer grounded in the gold context>",
  "gold_context": "chunk-118",
  "hard_negatives": ["chunk-117", "chunk-203"],
  "answerability": "answerable",
  "checked_by": "human"
}

That last arrow is the one teams skip. A synthetic set has an expiry date tied to the knowledge base, and the drift argument that applies to production documents applies to the test data just as much: static sets go stale almost immediately. In CI, gate on relative numbers, a drop in recall at 5 or in faithfulness against the previous run, and fail the build only when the human slice agrees that something really changed. The surrounding plumbing, thresholds and CI wiring, is the subject of LLM evals for product features, and Ragas, DeepEval and Red Hat's SDG Hub are the three generators I reach for first.

Sources

1. Red Hat Developers: Synthetic data for RAG evaluation: why your RAG system needs better testing (Feb 23, 2026)
2. Ragas docs: Testset Generation for RAG
3. Ragas docs: Testset generation for RAG, query types and scenarios
4. NVIDIA Technical Blog: Evaluating and Enhancing RAG Pipeline Performance Using Synthetic Data (Apr 7, 2025)
5. DeepEval docs: Datasets and the Synthesizer
6. Rajpurkar, Jia, Liang: Know What You Don't Know: Unanswerable Questions for SQuAD (ACL 2018)
7. Rahmani et al.: Towards Understanding Bias in Synthetic Data for Evaluation (CIKM 2025)
8. Cheng et al.: A Survey on Data Contamination for Large Language Models (2025)
9. Xu et al.: Infini-gram mini: Exact n-gram Search at the Internet Scale (EMNLP 2025)
10. Panickssery, Bowman, Feng: LLM Evaluators Recognize and Favor Their Own Generations
11. Enevoldsen et al.: MMTEB, Massive Multilingual Text Embedding Benchmark (ICLR 2025)
12. Evidently AI: How to align an LLM judge with human labels (Nov 14, 2025)